PLoS ONE, Open Access, and the Future of Scholarly Publishing 


Open Access (OA) advocates argue that PLoS ONE is now the largest scholarly journal in the world. 
Its parent organisation — Public Library of Science (PLoS) — was co-founded in 2001 by Nobel 
Laureate Harold Varmus. What does the history of PLoS tell us about the development of PLoS 
ONE? What does the success of PLoS ONE tell us about OA? And what does the current rush by other 
publishers to clone PLoS ONE tell us about the future of scholarly communication? 


The most important thing is to find out what is the most important thing 
Shunryu Suzuki 


Our story begins in 1998, in a coffee shop located on the corner of Cole and Parnassus in San 
Francisco. It was here, Harold Varmus reports, that the seeds of PLoS were sown, during a seminal 
conversation he had with colleague Patrick Brown.' Only at that point did Varmus realise what a 
mess scholarly communication was in. Until then, he says, he had been “an innocent person who 
went along with the system as it existed’. 





Enlightenment began when Brown pointed out to Varmus that when scientists publish their papers 
they routinely (and without payment) assign ownership in them to the publisher. Publishers then 
lock the papers behind a paywall and charge other researchers a toll (subscription) to read them, 
thereby restricting the number of potential readers. 


Since scientists crave readers (and the consequent “impact”) above all else, Brown reminded 
Varmus, the current system is illogical, counterproductive, and unfair to the research community. 
While it may have been necessary to enter into this Faustian bargain with publishers in a print 
environment (since it was the only way to get published, and print inevitably restricts readership), 
Brown added, it is no longer necessary in an online world — where the only barriers to the free-flow 
of information are artificial ones. 





Physicists, Brown said, have overcome this “access” problem by posting preprints of all their papers 
on a web-based server called arXiv. Created by Paul Ginsparg in 1991, arXiv allows physical 
scientists to ensure that their work is freely available to all. “Should not the biomedical sciences be 
doing something similar?” Brown asked Varmus. 





It was doubtless no accident that Brown — who had previously worked with the Nobel Laureate — 
chose Varmus as his audience for a lecture on scholarly publishing: at the time Varmus was director 
of the National Institutes of Health (NIH) — the largest source of funding for medical research in 
the world. He was, therefore, ideally placed to spearhead the revolution that Brown believed was 
necessary. 


Fortunately for the open access movement (as it later became known) Varmus immediately grasped 
the nature of the problem — aided perhaps by some residual Zen wisdom emanating from the walls 
of the coffee shop they were sitting in, which had once been the Tassajara Bakery.” Varmus 
emerged from the café persuaded that it would be a good thing if publicly-funded research could 
be freed from the publishers’ digital padlocks.*? And he went straight back to the NIH to consult with 
colleagues to that end. 





Again fortuitously, one of the first people Varmus broached the topic with was David Lipman — 
director of the NIH-based National Center for Biotechnology Information (NCBI). NCBI was home to 
the OA sequence database GenBank, and Lipman was an enthusiastic supporter of the notion that 
research should be freely available on the Web. By now Varmus’ conversion was complete. 


' Patrick O Brown is a geneticist at Stanford University School of Medicine. 

* The Tassajara Bakery was a Zen Center venture promoted by Richard Baker, and an extension of the baking 
practices at Tassajara Zen Mountain Center. It was sold off in 1992 and eventually closed, presumably then 
becoming the café in which Varmus and Brown met. 

* In his book The Art and Politics of Science Varmus heads the section describing his meeting with Brown as 
“My Conversion: The Making of a Radical”. p. 255. 
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This conversion was to see Varmus embark on a journey that would lead to the founding of a new 
publisher called Public Library of Science, the launch of two prestigious OA journals (PLoS Biology 
and PLoS Medicine), and subsequently to the creation of what OA advocates maintain is now the 
largest scholarly journal in the world — PLOS ONE. 


Varmus’ first attempt to liberate biomedical research from behind publisher paywalls took place 
during his tenure at the NIH, when he proposed a service called E-Biomed. Varmus envisaged E- 
Biomed as being “a huge digital library in which we would place existing journals, plus possibly 

some new journals, and then make it all fully searchable and available at the click of a mouse.” 


As we shall see, Varmus’ journey was to prove no walk in the park, and some believe his project 
lost its bearings on the way. Rather than providing a solution, they argue, PLoS may have become 
part of the problem. 


Certainly PLoS ONE has proved controversial. This became evident to me last year, when a 
researcher drew my attention to a row that had erupted over a paper the journal had published on 
“wind setdown”. Even some of the journal’s own academic editors appeared to be of the view that 
the paper should not have been published (in its current form at least). As the row appeared to 
raise questions about PLoS ONE’s review process — and about PLoS ONE more broadly — | 
contacted PLoS ONE executive editor Damian Pattinson. 





The response | got served only to pique my interest: While Pattinson invited me to send over a list 
of questions, | subsequently received an email from PLoS ONE publisher Peter Binfield informing me 
that it had been decided not to answer my questions after all. 


Explaining the decision Binfield said: “We’ve given this more thought, and I’m afraid that we don’t 
wish to engage in the long Q&A you have proposed. You have raised many questions now about this 
one paper along with various broader questions about PLoS ONE. | don’t think we have anything 
further to say about the article at this point, and so it doesn’t seem appropriate to use the 
discussion that surrounds this article as a way to build a much more extensive discussion about PLoS 
ONE.” 


He did however attach a statement addressing one of the questions | had asked. (See page 22) 

| took Binfield’s position to be that the row over the wind setdown paper raised no larger questions 
about PLoS ONE, its review process, or PLoS more generally. If that’s right, | disagree, and | hope | 
can demonstrate why in what follows. 

But even if Binfield were correct, one must wonder why a non-profit organisation that exists 
courtesy of the public purse, and which presents itself as an open and transparent organisation, 


should decline to answer questions sent to it by a journalist. 


That | found odd. 


Plan A: Advocacy and activism 


But first let’s back up to 1998 again. As noted, the initial outcome of Varmus’ conversation with 
Brown was a proposal for E-Biomed — an outline of which Varmus published in May 1999. 


At first glance, E-Biomed appeared to be a biomedical version of arXiv, and would include a variety 
of un-refereed material, including preprints, data sets and “conversations with authors of 
previously published papers.” 





* As Varmus puts it in his book (p. 256), “E-Biomed would also display, in a separate section, reports that were 
not peer-reviewed in the traditional manner, including papers that might normally not be published at all, such 
as those describing a new method or idea, or those reporting negative results from experiments or clinical 
trials.” 
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On closer examination, however, it was clear that the proposal went further: arXiv assumes that 
papers posted in it will subsequently be peer reviewed and published in a traditional journal.? It 
was envisaged that E-Biomed would review and publish (online) at least some of the papers itself. 


As the proposal explained, content would be posted into E-Biomed in two ways: "(i) Many reports 
would be submitted to editorial boards. These boards could be identical to those that represent 
current print journals or they might be composed of members of scientific societies or other groups 
approved by the E-biomed Governing Board. (ii) Other reports would be posted immediately in 

the E-Biomed repository, prior to any conventional peer review, after passing a simple screen for 
appropriateness." 


Naively, Varmus failed to appreciate that his plan would prove controversial. So far as he was 
concerned he was simply proposing a way of using the Internet to improve scholarly 
communication. 


More politically astute colleagues at NIH, however, were startled, both at the audacity of his 
proposal, and the way in which he was going about things. As one insider put it to me in 2006, 
“Varmus wrote the proposal himself and just sent it out to a bunch of people as a rough draft. For 


x,” 


us it was like ‘Oh, my God: he clearly doesn’t know what he is getting into’. 


The truth of this became apparent almost immediately. To Varmus’ surprise, reaction was loud, 
visceral and highly critical. He was accused of trying to dispense with peer-review, and of seeking 
to destroy traditional journals. Critics also complained that he was planning to organise “a take- 
over by the US government of an activity that should be international in character and belong in 
the private sector.” 


Varmus later joked to New Scientist, “I must have known that | was not going to be at NIH for much 
longer, because this caused a tremendous political argument: what the hell was | trying to do to 
destroy the publication industry.” 


Opposition was so virulent, and so vocal, that by the time Varmus’ initiative saw the light of day it 
was a pale shadow of what he had planned. Rob Kling et al. were later to characterise the process 
in this way: "[I]n less than a year, the E-Biomed proposal was radically transformed, eliminating the 
preprint section, instituting delays between article publication and posting to the archive, and 
changing the name to ‘PubMed Central’.” 


This, King added, was a consequence of "highly visible and highly influential statements made by 
publishers and scientific societies against the proposal." 


PubMed Central was eventually launched in February 2000. 


But however naive he may be, Varmus is not a man to give up. Within a year of leaving NIH he had 
co-founded — with Brown and Michael Eisen® — the advocacy organisation Public Library of 
Science, and launched a new front in his campaign to free the refereed literature. 


Initially PLoS focused on trying to strong-arm publishers into accepting OA, and to that end in 2001 
an open letter was published that read: “We support the establishment of an online public library 
that would provide the full contents of the published record of research and scholarly discourse in 
medicine and the life sciences in a freely accessible, fully searchable, interlinked form." 


Members of the research community were invited to sign the letter, and in doing so to make the 
following commitment: “[W]e pledge that, beginning in September 2001, we will publish in, edit or 
review for, and personally subscribe to only those scholarly and scientific journals that have agreed 
to grant unrestricted free distribution rights to any and all original research reports that they have 
published, through PubMed Central and similar online public resources, within 6 months of their 
initial publication date.” 





> Although, as we shall see, it does have a “moderation” process. 
° Bisen is an associate professor of genetics, genomics and development at the University of California. 
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Initial response to the letter was extremely encouraging: Within a short space of time nearly 34,000 
scientists from 180 different countries had signed it. 


But while a handful of publishers complied with the demands outlined in PLoS’ letter, most blithely 
ignored them. Even more discouraging, most of the scientist signatories proved themselves happy 
to forswear their own pledge and continued publishing in the very journals that had turned a deaf 
ear to them. 


Plan B: If you can’t beat them ... 


In retrospect it’s clear that the wider research community wasn’t ready for the kind of bottom-up 
revolution that physicists had embarked upon. So Varmus and his colleagues had again to re-group. 


Fortunately, in December 2002 the Gordon and Betty Moore Foundation awarded PLoS a $9 million 
grant. This allowed Varmus and his colleagues to adopt a completely different approach, and 
reinvent PLoS as a publisher. Not any old publisher, but an open access publisher — one committed 
to making every paper it published freely available on the Web. 


To achieve this PLoS adopted a very different business model.’ As the Gordon and Betty Moore 
Foundation press release put it, “The PLoS journals will retain all of the important features of 
scientific journals, including rigorous peer-review and high editorial standards, but will use a new 
business model in which the costs of these services are recovered by modest fees on each published 
paper.” 


In other words, rather than charging readers (or their institutions) a subscription to read the 
papers, PLoS would charge authors (or their funders or institutions) a fee to publish them. Later 
dubbed the article process charge (APC), PLoS’ modest fee was set at $1,500 per article —a 
payment designed to defray the publisher’s costs, and so obviate the need to charge for access. 


Even more ambitiously, PLoS founders set out to compete with the world’s most prestigious 
subscription journals — journals like Nature, Science and Cell. 





PLoS Biology was launched in 2002, and PLoS Medicine the next year.® Writing in the first issue of 
PLoS Biology, the three founders explained: “Today, with the launch of PLoS Biology, we take ona 
new role as publishers, to demonstrate that high-quality journals can flourish without charging for 
access.” 


They added, “Our aim is to catalyse a revolution in scientific publishing by providing a compelling 
demonstration of the value and feasibility of open-access publication. If we succeed, everyone who 
has access to a computer and an Internet connection will be a keystroke away from our living 
treasury of scientific and medical knowledge.” 


In fact, creating OA journals is not difficult. The challenge lay in ensuring that PLoS became 
financially self-sufficient before its grant ran out, and so demonstrate that author-pays publishing 
is sustainable.’ 


About this there was no shortage of sceptics, particularly amongst publishers. In October 2003 
Nature’s Declan Butler argued that $1,500 per paper would never provide sufficient revenue to 
keep PLoS afloat. To support his argument he quoted Yale University cell biologist Ira Mellman, 
then editor of Rockefeller University Press’s Journal of Cell Biology, who said: “I feel that PLoS's 
estimate is low by four — to six fold”. 





7 PLoS was not the first to use the author-pays model. It was pioneered by OA publisher BioMed Central, a 
private company founded in 1998. 

5 Since then a number of other journals have been launched, but it is PLoS Biology, PLoS Medicine and PLoS 
ONE that have attracted most attention, and are the key products. 

* OA advocates object to the term “author-pays”. For that reason I shall use the term “author-side fees” from 
henceforth. 
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Of course, Nature had an axe to grind. Not only was PLoS by now poaching staff from the journal, 
but OA posed a potential threat to the subscription business that sustained Nature: By deciding to 
charge $1,500 per paper, for instance, PLoS had breathed new life into a long-standing belief that 
scholarly publishers make excessive profits from the research community (and thus the taxpayer, 

who funds most research). 


This was evident a year later, when Nature gave evidence to a UK Science & Technology Select 
Committee enquiry into scientific publishing. Richard Charkin, then CEO of Macmillan (owner of 
Nature Publishing Group’s), told British MPs that it costs Nature $10,000-$30,000 to publish a 

1 
paper. 








British politicians could not help but wonder: If PLoS can make ends meet by charging $1,500, then 
how could incumbent publishers justify levying a subscription that, they said, assumed a per-article 
cost of $10,000 to $30,000?" 


Consequently Nature was keen to convince the world that PLoS would not survive if it did not 
increase its prices substantially. Butler returned to his theme again in June 2006, reporting that an 
analysis of PLoS’ accounts “shows that the company falls far short of its stated goal of quickly 
breaking even.” 


He added triumphantly, “In an attempt to redress its finances, PLoS will next month hike the charge 
for publishing in its journals.” 


Butler was right on that point: shortly afterwards PLoS announced that it was raising its APC to 
$2,500. 


But Butler still believed that PLoS would struggle to make ends meet. “The figures show that PLoS 
lost almost $1 million last year,” he wrote. “Moreover, its total income from fees and advertising 
currently covers just 35% of its total costs. And although this income is increasing — from $0.75 
million in 2003-04 to $0.9 million in 2004-05 — it lags far behind spending, which has soared from 
$1.5 million to around $5.5 million over the past three years.” 


Were Varmus and his colleagues about to prove themselves naive again? Had they significantly 
underestimated the costs associated with running a publishing house, particularly when competing 
with the world’s most prestigious journals? 


Or were they simply being canny, conscious that it is better to win the hearts and minds of 
customers before handing them the bill? 


Those who had been following Varmus’ OA activities since 1998, however, may have been more 
inclined to conclude that by creating traditional peer-reviewed journals (OA or otherwise) PLoS had 
lost its way, or even sold out. 


PLoS ONE: Return to first principles? 


Shortly afterwards, however, PLoS announced the launch of PLoS ONE. This was to be very different 
to PLoS Medicine and PLoS Biology, and has served to change the nature of the debate — both 
about PLoS and about OA. While some saw PLoS ONE as a return to first principles others were to 
take a more cynical line. 


The press release announcing the launch of the new journal described it in this way: “[VJirtually 
everything about PLoS ONE is new: the peer-review strategy, the production workflow, the author 
experience, the user interface, and the software that provides the publishing platform.” 


'° As Charkin put it to the UK Science & Technology Select Committee, “Very crudely, £30 million of sales: 
we get income of £30 million and we publish 1,000 papers a year.” 

'' This figure is sometimes quoted in pounds rather than dollars. See for instance here. 
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What was most distinctive about PLoS ONE at first glance was its scope. As Varmus put it in his 
book, PLoS ONE was to be “encyclopaedic”."* This was radical because the long-standing trend in 
scholarly publishing had been for journals to become increasingly specialised and narrower in 
scope, not broader.'? PLoS ONE, however, planned to accept papers from all disciplines within 
science and medicine. 


Speaking to me in 2006 the then managing editor of PLoS ONE Chris Surridge pointed out that there 
was in fact no scientific topic that PLoS ONE would not consider. As he put it, “[I]f someone came 
to PLOS ONE with nuclear physics we would welcome them with open arms, and go and find some 
editors who knew something about nuclear physics who could make a decision about whether the 
paper was worth publishing. And if they felt it was worth publishing we would then publish it.”"* 


In retrospect, however, it is PLoS ONE‘s approach to peer review that has proved most significant. 
Varmus hinted at this when | spoke to him in October 2005. PLoS ONE, he said, was envisaged as, “a 
very large compendium of papers that have been vetted for scientific quality, but which will not be 
confined in terms of their likely importance.” 


Surridge expanded on this when he spoke to me at the time of PLoS ONE’s launch. “Traditionally a 
lot of the work that goes into peer reviewing consists of asking questions like: ‘How significant is 
this? How surprising are the conclusions?’ Essentially, these are subjective questions. A more 
objective question to ask would be: ‘Is this properly done science?” 


To that end, Surridge added, PLOS ONE reviewers “will be asked to answer a simpler question than 
has traditionally been asked. Essentially, he added that question is: ‘Has the science in this paper 


been done well enough to warrant it being entered into the scientific literature as a whole’. 
As a consequence, he added, “[W]e are looking at an easier decision on whether to publish.” 


It is important to note that telling reviewers to ask a simpler question was based on PLoS’ belief 
that a scientific paper’s significance can only be properly established after publication. “We 
believe that the more subjective questions about how a paper relates to other work, and where it 
fits into the whole corpus of scientific literature are still important questions,” said Surridge. “But 
we feel that these can be better answered via an open peer review process that takes place after 
the paper has been published.” 


In other words, in introducing a simpler review process PLoS ONE assumed that reviewing of the 
papers would continue after publication, and would be done by the wider research community. As 
the launch press release put it, “All readers will have the tools to add comments, annotations, and 
ratings to each article, so that post-publication review forms an integral part of the review 
process.” 


It added, “PLOS ONE will empower the scientific community as a whole to engage in an open 
discussion on every piece of published work, capturing the varied and extremely valuable 
assessment of published papers that occurs after the work has been published.” 


We shall come back to that point. 
Meanwhile, for authors, Surridge told me the main attraction of PLoS ONE would be its ability to 
maximise the number of people (eyeballs) able to read their work, while at the same time 


providing a quicker, simpler, more efficient publication process. 


As he put it: “If you want to spend months revising and re-revising your paper to satisfy the 
demands of two or three experts who may or may not be the most appropriate people to judge your 





' On p. 264 of his book Varmus describes PLoS ONE as an, “encyclopaedic, high volume publishing site”. 

'8 Indeed, the history of the scholarly journal (mapping the development of science in general) has been one of 
increasing specialisation. 

' Today PLoS ONE publishes papers in subjects that range from biochemistry to medicine, to ecology, 
computer science, and the marine and aquatic sciences. 
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work (and so face a small chance of eventually being published), and if you are happy for your 
paper to be inaccessible to many of the readers who might wish to read it, then conventional 
journals are for you.” 


If, on the other hand, he added, “you want to share your results — as soon as possible — with the 
whole scientific community in a way designed to stimulate discourse and so scientific advance, 
then PLoS ONE will be your journal of choice.” 


It was an exciting and logical move on the part of PLoS founders. Importantly, the new journal 
appeared to fit well with Varmus’ initial vision. It also held out the promise of rapidly increasing 
the number of papers freely available on the Web. 


This last point was important because it had become clear that highly-selective subject-specific 
journals like PLoS Biology and PLoS Medicine would not be able to catalyse PLoS’ revolution on their 
own — not least because in order to compete with high-quality prestigious subscription journals 
like Nature they were having to reject around 80% of the papers submitted to them. 





As a PLoS Medicine editorial explained in 2006, “We feel that the unwanted side-effect of this 
rejection rate — and the authors of rejected papers would agree — is that we are turning away 
many valuable contributions. 


It added, “As an organisation whose vision is a world of scientific publishing where there is an open- 
access journal for every paper worth publishing, it is essential that PLoS continues to create 
additional open-access venues to help reach this goal.” 


It was for this reason, the editorial pointed out, that PLoS ONE had been launched. The aim, it 
seems, was to flood the world with OA papers — an ambition that was to become something of an 
idée fixe for PLoS employees. 


So was PLoS ONE a return to first principles? It was, after all, an arXiv-like service (Varmus’ “huge 
digital library”), although unlike arXiv (but envisaged with E-Biomed) it would review papers itself 
rather than leave the task to others. 


It is worth pointing out that arXiv does itself operate a “moderation” system. It assumes, however, 
that the papers will subsequently be published in traditional journals. PLoS ONE’s review process 
was therefore (in theory at least) more traditional than arXiv’s moderation process’. Moreover, it 
planned to send most papers out for review rather than moderate them internally in the manner of 
arXiv. Unlike both arXiv and E-Biomed, however, PLoS ONE planned to publish papers in all 
disciplines within science and medicines”'* — so its ambitions were broader. 


Certainly when he spoke to me in 2006 Surridge appeared keen to portray PLoS ONE as more in 
keeping with PLoS’ founding mission than PLoS Medicine and PLoS Biology. 


As he put it, “[T]he system of disseminating scientific research has become extremely inefficient, 
and the concept of the journal has been eroded by the Internet ... [SJimply publishing journals and 
distributing them electronically isn’t using the full potential of the Web to make the dissemination 
of scientific information efficient and effective. It just isn't the best way to do it anymore.”"” 


Sceptics, meanwhile, saw a more self-serving motive behind PLoS ONE. As Butler had pointed out, 
competing effectively with journals like Nature and Science would require more money than PLoS 
had access to. Cynics concluded, therefore, that PLoS ONE’s broad disciplinary approach and 
simplified peer review was an attempt to create a cash cow. 





'S arXiv does not peer review the papers submitted to it but, as we note, it does have a moderation system. 
'© Tt is worth noting that the scope of arXiv too has grown over time. Today it covers Physics, Mathematics, 
Computer Science, Quantitative Biology, Quantitative Finance and Statistics. 

'7 We should note, however, that PLoS ONE does describe itself as a journal, where many view it more as a 
database, or archival service. 
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After all, they reasoned, PLoS ONE planned to charge $1,250 per paper (subsequently increased to 
$1,350) for providing, by its own admission, a no-frills stripped-down service. As such, they viewed 
PLoS ONE as a mass market product, but one designed to operate on high margins. 


Even if that were true, however, PLoS ONE would first need to demonstrate that it was sufficiently 
appealing to the research community to prove successful. In 2006 success was by no means certain. 


Bulk, cheap publishing 
Certainly some observers had doubts about PLoS ONE. Jason Kelly, for instance, wondered how the 


journal expected to “build up a community of editors/reviewers/annotators?” For instance, he 
asked: “What is the incentive for scientists to take the time to provide this feedback?” 





As we shall see, Kelly’s scepticism on this last point was to prove justified. However, always keen 
to bulk up their CVs, researchers were soon signing up as editors, and submitting papers, to PLoS 
ONE in droves. Today the journal has over 1,000 academic editors, and the number of papers 
submitted is growing exponentially. 


Consider that in 2006 PLoS ONE published just 138 articles. By 2007 this had grown to 1,230 and to 
2,716 in 2008, making PLoS ONE the largest open access journal in the world. In 2009 it published 
4,405 papers (a 65% increase on 2008), taking the total since launch to 8,495. 


And this rapid growth continued in 2010. By the end of the year PLoS ONE had published 6,749 
papers (a further 50% increase), out-publishing both Physical Review B and Applied Physics Letters. 








As a consequence, OA advocates now boast that PLoS ONE is the largest journal in the world. As it 
was only launched in 2006 it is not in fact the largest journal in terms of total numbers of papers 
published (rather than annual growth). But to have published 15,238 papers in a little over four 
years is a significant achievement. And at the time of writing this figure had grown to 17,099, up 
1,861 on the year. 


Why has PLoS ONE proved so successful? It surely helped that by the time the journal was launched 
PLoS as an organisation had spent a great deal of money marketing itself as an upstart right-on 
publisher willing and able to prove a David to the Goliaths of the industry. It doubtless also helped 
that PLoS was co-founded by a Nobel Prize winner and former director of NIH. 


And as more and more authors concluded that having a paper published in PLoS ONE was sexy, so 
the journal entered a virtuous circle. This was to see it win the Association for Learned and 
Professional Society Publishers’ Publishing Innovation Award in 2009 — for being “bold and 
successful and shaping the future of publishing" — and gain its first Impact Factor (IF) in 2010, 
providing authors with an even more compelling reason to publish in the journal. 





For traditionalists, PLoS ONE’s IF came as a shock. Describing it as “a stunning 4.351”, OA critic Phil_ 
Davis wrote on the Society for Scholarly Publishing-sponsored blog The Scholarly Kitchen: “[T]here 
seems something very strange about a journal that accepts nearly 70% of all submissions yet 
achieves such a score, especially with its first assessment.” 


OA advocates, however, nodded wisely and asserted that it was a good demonstration of the so- 
called “open access advantage”. '® This was a rather unsatisfactory explanation for Davis — not just 
because he is a long-time critic of OA, but because his PhD thesis concluded that the OA advantage 
is a chimera. 


But doubtless the fundamental appeal of PLoS ONE was exactly as Surridge had predicted: a journal 
that publishes nearly 70% of the papers submitted to it is a godsend for ambitious researchers 
looking for an outlet for their papers. That they can do so without having to spend months revising 





'S By making their papers freely available on the Web, proponents of the open access advantage argue, 
researchers can expect to attract a far greater number of citations, and thus have a greater impact. 
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and re-revising their papers to satisfy the often quixotic demands of anonymous reviewers — “and 
so face a small chance of eventually being published” — made PLoS ONE all the more attractive. 


For PLoS itself the most obvious benefit of the new journal’s success was that claims that it was not 
financially sustainable began to look increasingly hollow — for as submissions to PLoS ONE climbed 
so did PLoS’ revenues. 


In 2010 PLoS was able to announce that it was well on the way to self-sufficiency. A progress 
update published last July, for instance, reported that PLoS “continued on its path towards 
operating profitability in 2009, exceeding plan expectations on all fronts”. 


Specifically, total revenues for the period grew to $8.8 million — a 36% increase over 2008 levels. 
This, PLoS explained, had been “fuelled mainly by strong growth in publishing volumes.”” 


But while the sudden inflow of income was good news for PLoS, critics merely used it as another 
stick with which to beat the publisher. And a new charge emerged: PLoS’ primary aim, they argued, 
was not that of “shaping the future of publishing”, but keeping itself afloat at all costs. 


In another article published in July 2008, Butler suggested that in order to make ends meet PLoS 
had decided to go down market, “relying on bulk, cheap publishing of lower quality papers to 
subsidise its handful of high-quality flagship journals’. 





Entitled, “PLoS stays afloat with bulk publishing” ”°, Butler’s article set off a firestorm of criticism 
and moral outrage amongst OA advocates. “This clumsy hatchet job from Nature reporter Declan 
Butler is beneath him, a poor excuse for journalism and an affront to the respect with which many 
of his colleagues are regarded by the research community,” bridled then PLoS employee Bora 
Zivkovic. 


“With this scurrilous parroting of anti-OA FUD, Nature makes pretty clear where its interests and its 
allies are,” snorted Bill Hooker. 





Costs 


The backlash from OA advocates was sufficiently fierce that Nature’s Timo Hannay felt compelled 
to join the discussion. Rather than seek to placate OA advocates, however, he repeated Butler’s 
claims, and then added another charge: PLoS, he said, was guilty of anti-competitive behaviour. 





While stressing he was not claiming that PLoS was being intentionally anti-competitive, Hannay 
argued that the publisher had nevertheless created “effects that run directly against its stated aims 
of spreading the open access model across all of science publishing.” 


He elaborated: “The situation back in 2002 was that existing publishers (Nature Publishing Group 
among them) were very sceptical that author-pays economics could ever work for high-end 
journals. Fast-forward to 2008 and far from bringing other publishers along with them, PLoS has 
helped to confirm these suspicions.” 


Echoing Butler, Hannay added that PLoS’ flagship journals had only survived courtesy of subsidies, 
first from the Gordon and Betty Moore Foundation, subsequently from PLoS ONE’s bulk-publishing 





'° The update added: “Consistent with our sustainability strategy to achieve operating profitability in 2010, 
operating revenues increased 45% over 2008 levels ($8.9MM) while expenses increased only 24% over the 
same period, substantially narrowing our operating gap. Public support for the year was $0.5MM, slightly lower 
than the previous year but consistent with expectations as we fund more of our growth through our publishing 
operations. PLoS posted its first profitable quarter in Q1 2010 due to strong growth in publishing activity, and 
we anticipate meeting or exceeding our financial targets for 2010.” The 2009 figures are available on GuideStar 
here 

°” The term “bulk publishing” was doubtless coined for PLoS ONE in order to attack it. We should note, 
however, that in his book Varmus himself describes PLoS ONE as “a high volume publishing site”. (p. 264) 
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activities. And by cross-subsiding its journals, he concluded, PLoS had blocked other publishers 
from the market. 


PLoS, said Hannay, “has created heavily and perpetually subsidised journals that act as a strong 
disincentive for anyone else to try a similar publishing model. Which other publisher will enter the 
high-end author-pays segment now that there is an incumbent with ‘private income’ that will always 
be able undercut any competitor who needs to be economically self-sustaining? In short, the effect 
has been to reduce competition in an industry that, for the good of science as a whole, ought to 
have more competition.” 


Hannay concluded with a barbed “hypothetical” scenario: “[YJou might imagine setting up a small 
series of loss-making open-access journals, thus killing off any chance of a competitive market 
emerging in that segment. You'd fund them through donations for as long as possible, but such 
income is vulnerable to the vagaries of donors so eventually you'd have to cross-subsidise them 
from a profitable business line. And all the while you'd explain that you're doing the world a 
favour.” 


While critics preferred not to labour the point, in claiming that it was subsidising its sister journals 
they were, by implication, also accusing PLoS ONE of overcharging authors.”' For perhaps self- 
serving reasons, they preferred to construct a narrative that said PLoS ONE was generating excess 
revenue by providing low-cost, low-quality peer review services. 


But however the narrative was nuanced, it encompasses two intersecting claims: First, in order to 
subsidise its flagship journals, PLoS ONE’s APC of $1,350” had been set higher than warranted for 
the service provided (a cost issue). Second, PLoS ONE was able to do this by providing minimal, 
possibly inadequate, peer review (a quality issue). 


The underlying premise to this double claim is that the more papers a journal publishes the lower it 
will have set its bar of scrutiny, and thus the cheaper it will be to publish a paper. Or to put it the 
other way round: The more papers a journal rejects the higher will be the quality bar, and the 
more costly each paper will be to publish — because publishers incur costs not just for the papers 
they accept, but for those they reject too.” 


While this argument is widely accepted it is a very crude one, and certainly open to challenge. 
Suffice it to say that Nature’s criticism of PLoS ONE was postulated on the claim that if a high- 
quality journal (like Nature) rejects 90% of the papers submitted to it, then the cost of reviewing 
100 papers will — for accounting purposes — have to be borne by the 10 accepted papers.” If, by 
contrast, a less picky journal (like PLoS ONE) rejects only 30% of the papers submitted to it, then 
the cost of reviewing 100 papers can be distributed amongst the 70 accepted papers — significantly 
reducing per-paper costs.” 





In the traditional subscription publishing environment the relationship between rejection rates and 
per-paper costs was not overly dwelt upon, or much discussed — not least because publishers did 
not feel compelled to justify their costs. When authors are asked to pay to publish their papers, 





*! Why would they? It is not in the interests of subscription publishers to draw attention to the research 
community’s fears that scholarly publishing can be extremely lucrative. 

~* Initially $1,250. 

3 In a footnote on p. 264 of his book Varmus says, “Since PLoS Biology generally publishes fewer than twenty 
articles a month, the rejection rate is high; this factor both ensures prestige and drives expenses up, since well- 
paid professional editors must monitor the review process for rejected as well as accepted papers.” 

* As Nature put it in 2004: “Economics dictates that high-quality journals like Nature have a high unit cost per 
paper published, because for every article published more than ten have been reviewed and deselected.” 

> It is worth pointing out that most papers in high-quality journals are not actually sent out for review. A Nature 
editor recently confessed, for instance, that he rejects “at least four in every five manuscripts straight off the bat, 
before review.” How much does it cost a publisher to hire someone to throw most of the papers it receives 
straight into the bin? In this light one is inclined to be highly sceptical of Nature’s claim that it costs $10,000 to 
$30,000 to publish a paper. 
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however, per-article costs become much more significant (and in theory transparent). Certainly 
they are thrown into sharp relief, and inevitably invite questions about costs not previously asked. 


What none of this tells us, of course, are the actual costs the publisher of a journal incurs in 
publishing a paper — since even in an OA environment what the author pays may be entirely 
unrelated to the costs incurred by the publisher (which is what, by implication, PLoS ONE critics 
argue). So a key question to ask is, does PLoS ONE really incur the circa $1,000 worth of processing 
costs for each paper implied by the above argument; or is it, as critics claim, overcharging authors 
in order to subsidise its sister journals? 


In this respect, it is worth comparing the cost of PLoS ONE’s APC with the costs that arXiv estimates 
it incurs for every article posted to its database. Last year Cornell’s Simeon Warner reported that 
this has been calculated at $7 per paper. 


As noted earlier, while arXiv does not operate a formal review process like PLoS ONE, it does 
nevertheless subject all papers to a process of moderation. In other words, arXiv is not simply a 
dumb repository into which documents can be thrown willy-nilly; all the papers submitted are 
checked — for suitability, appropriateness, format, and what arXiv calls “refereeable quality” 
amongst other things. If any document submitted does not meet the necessary criteria it is 
removed. 


A relevant question to ask, therefore, is: To what extent does arXiv’s moderation process differ 
from the reviewing done by PLoS ONE, and is this difference reflected in the estimated costs 
incurred. Put simply, is the additional work undertaken by PLoS ONE 193 times more costly than 
arXiv’s moderation process?”° 


We should also note that while publishers casually boast about the costs they incur in “doing peer 
review”, all they actually do is manage the review process. The reviewing itself is done by the 
research community, without charge to the publisher. 


So is PLoS ONE overcharging in order to subsidise its sister journals? In the hope of finding out one 
of the questions | emailed PLoS ONE was: “What are the costs per paper for publishing in PLOS 
ONE?” 


Unfortunately, | received no reply to my question. 


The suspicion is that it is overcharging. And if it is, then Nature’s estimate that it incurs costs of 
between $10,000 and $30,000 to publish a paper begins to look unreasonably high.”” 


There is another cost issue we will need to consider later. That is, even if publishers’ charges were 
demonstrated to be fair and reasonable, can the research community still afford them, particularly 
at a time of global financial crisis? 


Quality 


We will come back to costs but, as indicated, the issue of costs is bound up with that of quality. 
This is a point Butler made in 2008 when he said that PLoS’ financial future looks brighter, “thanks 
to a cash cow in the form of PLoS ONE”. One, he added, that “uses a system of ‘light’ peer-review to 
publish any article considered methodologically sound.” 


Davis made a similar point last June on The Scholarly Kitchen. Arguing that PLoS ONE had been 
launched to fund the other PLoS journals he said, “In financial terms, the purpose of PLOS ONE is to 
subsidise the cost of publication for their two flagship journals, PLoS Biology and PLoS Medicine, 
two highly-selective, high-cost journals that would be unable to continue in their current state 


°° A 2009 JISC-funded report estimated that a “self-archiving with overlay services” OA model would allow 
savings of £260 million per annum in the UK alone. 

°7 As claimed here. 

11IPLOS ONE, Open Access, and the Future of Scholarly 
Publishing 


without financial subvention. PLoS ONE is able to do this because of low selectivity — rejecting 
only 3 out of every 10 manuscripts.” 


As noted, the assumption behind this claim is that the more papers a journal accepts the lower will 
be the quality of those that it publishes. That PLoS ONE states publicly that its reviewers are told 
to make an easier decision obviously fuels such claims. 


Kent Anderson, publisher of The Journal of Bone and Joint Surgery and another frequent blogger on 
The Scholarly Kitchen is a sceptic too. And he worries about the impact PLoS ONE could have on 
scholarly communication at large. Last April he reminded readers that PLOS ONE publishes 70% of 
the papers it receives. Describing it as “not a tight filter,” he added: “In its early days, PLoS had an 
opportunity to break free from the herd. Instead, it joined the herd. That’s poignant enough. Now, 
it’s potentially carrying a business model with PLoS ONE which could make people doubt the quality 
or safety of the herd.” 


The debate about PLoS ONE, therefore, is not just about the costs of publishing, and financial 
sustainability. It is also a debate about the quality of peer review. At bottom it is a question of 
whether the papers published by PLoS ONE are indeed of an acceptable standard to, in Surridge’s 
words, be entered “into the scientific literature as a whole.” If not, then what might be the likely 
impact on scholarly communication at large? 


So let’s look more closely at PLoS ONE’s peer review. Is it indeed “light” in the way Butler claimed? 
What does light peer review imply anyway, and what are the implications if PLoS ONE’s review is 
indeed light? 


We should note that PLoS has never described its peer review as light although, as we’ve seen, 
Surridge told me that when evaluating papers reviewers are instructed to ask a “simpler question” 
and make an “easier decision”. It would therefore seem fair to call the process light. PLoS has also 
never made any secret of the fact that it publishes the majority of papers submitted to it. Indeed, 
it prides itself on the fact. 


The key question, therefore, is whether PLoS ONE’s review process is felt to be adequate. 
Unfortunately, this is not an easy question to answer — for two main reasons. 


First, trying to assess the quality of peer review is very difficult because there are no universally 
agreed methods, standards, or formal processes against which to measure adequacy, or that specify 
how editorial decisions should be made. If one were to talk to ten different editors one would likely 
get ten different explanations of how peer review ought to be conducted, and exactly what is 
required. To a great extent, therefore, good peer review is in the eyes of the beholder 


Second, peer review is invariably a non-transparent process: reviewers usually remain anonymous 
and their reports are rarely made public. And while PLOS ONE initially trumpeted a commitment to 
openness and transparency, and promised to make reviewer reports publicly available, that 
commitment appears to have waned over time. Indeed, at some point there may have been a 
change of policy. That at least would seem to me to be the implication of a 2008 blog comment by 
PLOS ONE academic editor Cameron Neylon. 





When | emailed Neylon — a senior scientist at the Science & Technology Facilities Council, a UK 
funding body”® — and asked him to clarify his comment he replied: “I’m not sure it was so much a 
change in policy as a change in the practicalities. It was just proving to be too much work. However 
this happened before | became an AE. I’m pretty sure the hope is that the review process can be 
opened up to make that information available. But the best person to ask about the details would 
be Pete Binfield.” 


As we have seen, however, Binfield appears to be a little selective about what questions he 
answers, and by the time | emailed Neylon Binfield had in any case refused to answer the questions 
| had emailed to PLoS. 


*8 Neylon is also now editor-in-chief of the new BioMed Central OA journal Open Research Computation 
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In August 2010, however, Binfield did say something about transparency. Asked by The Scientist 
whether reviewers should reveal their identities he replied, “I think at some level there should be 
this transparency to the production process. The question is whether it puts off reviewers from 
being as frank with their comments, or even [from] review[ing a manuscript] in the first place.””’ 


Be that as it may, the best we can hope to do in terms of assessing the adequacy of PLoS ONE’s 
review process is to try and measure its practice against its stated standards and procedures. 


Let’s remind ourselves how PLoS ONE describes its review process. As we saw, PLoS ONE reviewers 
are told to ask a “simpler” question than when reviewing for traditional journals. That question, 
Surridge told me is: “Has the science in this paper been done well enough to warrant it being 
entered into the scientific literature as a whole?" 


We should note that this begs the question of how one defines “well enough”. 
As it happens, Surridge may subsequently have regretted describing the review process in the way 


he did when speaking to me. After the interview he wrote on the PLoS blog “I’m quite distressed to 
discover how inelegantly | speak. Richard was asking some tough questions and I’m no politician.” 





Others also picked up on Surridge’s discomfiture, and one commentator concluded “Perhaps there 
may have been some initial ambiguity about where the threshold for publication would be set.” 


This confusion might however be seen to support the view that PLoS ONE’s review process is 
problematic: If the managing editor of a journal struggles to explain how its review process works, 
and what standards are applied, will not reviewers be confused too? 


If we turn to the description of the PLoS ONE review process in the press release announcing its 
launch we read that: “[S]ubjective considerations like ‘likely impact,’ ‘degree of advance,’ or 
‘interest to a general reader’ will not play a role in deciding whether an article should be published 
or not.” 


The press release added, “Papers published by PLOS ONE will be held to rigorous standards of 
scientific quality.” 


The PLoS ONE web site describes the review process in this way: “Too often a journal's decision to 
publish a paper is dominated by what the Editor/s think is interesting and will gain greater 
readership — both of which are subjective judgments and lead to decisions which are frustrating 
and delay the publication of your work. PLOS ONE will rigorously peer-review your submissions and 
publish all papers that are judged to be technically sound.” 


We are bound to wonder how PLoS ONE’s “rigorous standards” are defined, and exactly what 
scientific quality and technical soundness are.*° We might also wonder if any potential 
contradiction arises from applying rigours standards while making a simpler decision — at least in 
the minds of reviewers. 


In any case, the key concepts for reviewers appear to the need to avoid making “subjective 
judgements”, to maintain “rigorous standards of scientific quality” and to ensure that papers are 
“technically sound”.”2" 





* Note that a recent BMJ paper concluded, “Telling peer reviewers that their signed reviews might be available 
in the public domain on the BMJ’s website had no important effect on review quality. Although the possibility 
of posting reviews online was associated with a high refusal rate among potential peer reviewers and an increase 
in the amount of time taken to write a review, we believe that the ethical arguments in favour of open peer 
review more than outweigh these disadvantages.” 

*° PLoS ONE’s Guidelines for Authors adds, “The research must have been performed to a technical standard 
high enough to allow robust conclusions to be drawn from the data.” But is this clearer? 

*! Elsewhere PLoS says of the PLoS ONE review process: “For PLoS ONE, peer review is handled by academic 
editors who seek the opinions of other relevant experts, but the question asked is only whether the submission 
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One is inclined to point out that interpreting these terms would in itself require making a 
subjective judgement. What exactly do they mean?” 


As it is, one can envisage reviewers scratching their heads, and then simply ignoring the guidelines 
all together. But with words like “easier” and “simpler” resounding in the minds they would hardly 
be likely to take a hard line., 


Authors not readers 


One frequently voiced concern about PLoS ONE’s review process is the journal’s insistence that 
reviewers make no judgement about the significance or importance of the research they are 
evaluating. The problem with this, they argue, is that busy researchers don’t have time to read 
everything published in their field. If the journals don’t alert them to the papers they really need to 
read then how can they keep up to date? 


As one researcher asked, “Is a faster publication time vs. no (immediate) research significance 
assessment of papers really a good trade-off? Faster publication is good for authors but not a must 
for readers.” 





This draws our attention to an important feature of “author-side” fees publishing: it radically 
changes the publisher’s perception of who the customer is. In an OA world journals inevitably focus 
much less on the needs of readers, and much more on the needs of authors. And in today’s “publish 
or perish” environment the priority for researchers (as authors) is to have as many papers published 
as possible. Only by doing so can they hope to advance their careers. 


But while a journal with a 70% acceptance rate is very attractive to authors, it is less so for the 
researcher-as-reader — because if (as claimed) high acceptance rates lead to a large number of 
low-quality papers then the filter problem will surely grow over time.*? 


So while other publishers have tended to focus on PLoS ONE’s business model, and the issue of 
pricing, for the research community the journal’s rapid “bulk publishing” model (what Varmus calls 
“high volume publishing’) raises concerns that the quality of published research will decline, and 
the task of identifying papers worth reading will be that much harder as a result. This implies a 
vicious circle rather than a virtuous one. And even some of PLOS ONE’s own academic editors have 
begun to worry about this. 


Meanwhile from the taxpayer’s point of view a bulk publishing model inevitably raises questions 
about both quality and cost e.g. how can we be sure that tax-payer’s money is being spent wisely? 
If the review process is lighter, how can the public be confident that when a researcher pays 
$1,350 to publish a paper he or she is getting value for money, and that the paper is being 
adequately vetted before being placed in the pubic domain? What exactly is being provided for 
taxpayer dollars?2° 


Finally, any discussion about PLoS ONE inevitably leads to the topic of publicity too. As we shall 
see, PLoS ONE papers are frequently used as marketing tools to promote not just the paper itself, 
but both PLoS and OA too — an activity that fosters exaggerated claims about the research in the 
paper and (ironically) its significance. The concern must be therefore that promoting research 





has been rigorously and ethically conducted and properly reported; no account is taken of the potential 
significance or target audience for the work.” 

* There is a slightly more detailed description on the PLoS ONE Web site, but I do not think this really 
addresses the ambiguities implicit in PLoS ONE’s definitions. 

°° The increasing focus on the author rather than the reader may also partly explain why journals are finding it 
far more difficult to find reviewers. See for instance the comments of this Nature editor: “over the past couple of 
months I have seen an increase in the incidences of referees declining to review manuscripts because they are 
too busy.” 

* On p. 264 of Varmus’ book he describes PLoS ONE as an, “encyclopaedic, high volume publishing site”. 

° As my experience suggests, Binfield is not of a mind to discuss this. 
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papers can hinder rather than aid the public understanding of science. And when the public is 
misled about the significance of published research it inevitably raises doubts about the underlying 
science, and thus the quality of PLoS ONE’s review process. 


But is there any evidence that PLOS ONE’s peer review is inadequate, or fails to meet its own 
standards? Can we be confident that the journal consistently demands “rigorous standards of 
scientific quality” from the papers it reviews? Can we be sure that it only publishes research that is 
“technically sound”? And can we be sure that its standards are in fact adequate? Finally, is there 
any reason to be concerned about the way in which PLoS ONE papers are promoted and presented 
to the public? 


Some have certainly expressed doubts on these matters. 
Drivel? 


In June 2007, for instance, PLoS ONE published a paper by a former investment banker on the 
relationship between male circumcision and AIDS. A press release written by the author publicising 
the paper stated: “male circumcision is found to be much less important as a deterrent to the 
global AIDS pandemic than previously thought ... [and] ... The new study finds that the number of 
infected prostitutes in a country is the key to explaining the degree to which AIDS has infected the 
general population.” 


It added: “prostitute communities are typically very highly infected with the virus themselves, and 
because of the large number of sex partners they have each year, can act as an engine driving 
infection rates to unusually high levels in the general population.” 


And it concluded that the study had “a number of important findings that should impact policy 
decisions in the future.” 


Shortly after the paper was published, however, AIDS researchers began to express a very different 
point of view, claiming that the paper was flawed in a number of important ways. 


Reporting on the controversy, Nature explained: “[A]ccording to several angry AIDS researchers the 
paper merely shows that the peer-review system of the journal that published it, PLOS ONE, failed 
on this occasion.” And added: “The study, they say, is flawed and, moreover, concerns a debate 
over statistical techniques that, in this instance, have been largely superseded by more powerful 
clinical trials.” 


Tim Farley, a Geneva-based official in the World Health Organisation's HIV-prevention team put it 
more bluntly, telling Nature, “The paper is total drivel, it should have been picked up in the review 
process.” He added, “In public health there are severe dangers in such stuff getting through.” 


In short, critics concluded that PLoS ONE’s review process had failed. They also felt that the paper 
had been inappropriately promoted by means of a press release put out by the author. True, a 
disclaimer from PLOS ONE had been attached to the press release, but as Farley pointed out, since 
the paper had been published in a peer-reviewed journal the views expressed in the press release 
inevitably acquired “a public perception of validity.” 


It did not help that the press release included the florid statement: “To argue we should do nothing 
about infected prostitutes during an AIDS epidemic because of a fear of creating a stigma against 
the infected would be like an animal rights activist claiming that a rabid dog must be allowed to 
run free in a neighbourhood regardless of how many men women and children he infected and 
killed.” 


Asked by Nature to comment Surridge responded, “There are lessons to be learned from all papers 
that we publish; we are a young journal ... We are feeling our way." 


Controversies like this clearly do not inspire confidence in PLoS ONE’s light review system. It is also 
striking that rather than producing its own press releases (although it does do that), PLoS ONE 


ISIPLOS ONE, Open Access, and the Future of Scholarly 
Publishing 


appears to prefer (and indeed encourages) authors to promote their papers by means of press 
releases, as apparently happened in this case. 


We might also be tempted to conclude that making self-serving claims about the importance or 
significance of a paper makes a mockery of PLoS ONE’s claim that these things can only be 
established after publication — and presumably by other researchers rather than the author. 


Of course one cannot generalise from a single paper, and an incident that took place nearly four 
years ago. However, a number of other PLoS ONE papers have subsequently attracted controversial 
attention, and there is growing suspicion that asking reviewers to make an “easier decision” on 
publication may be inherently problematic. 


Technically sound? 


Last April, for instance, Phil Davis drew attention to another PLoS ONE paper; a paper, he 
suggested, that cast some doubt on the journal’s review process. 


The paper was entitled “Do Pressures to Publish Increase Scientists’ Bias? An Empirical Support from 
US States Data”, and authored by the University of Edinburgh’s Daniele Fanelli. It was, said Davis, “a 
great example of a paper that may be technically fine, but fails miserably on theoretical grounds, 
and the methodology reveals it.” 





Davis was not the only one to express doubt: John Timmer raised a number of questions about the 
paper on ArsTechnica. Fannelli’s paper, Timmer explained, sought to test the hypothesis “that 
researchers in a competitive environment, who are most sensitive to the ‘publish or perish’ 
mentality that prevails in the sciences, would be less likely to publish papers that describe negative 
results.” 


While conceding that the issues discussed were problematic, Timmer concluded that the new data 
provided by the authors did not appear to offer an adequate explanation for their conclusion. “The 
authors themselves raise some significant questions about their interpretations without even going 
into some of the basic issues with the data,” he complained. *° 


Anderson also commented on the Fanelli paper. In a separate post on The Scholarly kitchen he 
concluded, “The study critiqued [by Davis] shoves a sloppy hypothesis into a forced positive.” 


Also critical was David Crotty — another habitué of The Scholarly Kitchen, and executive editor of 
Cold Spring Harbour Protocols. In a comment on Davis’ post Crotty suggested that it was not 
actually accurate to call the paper technically fine. “As you point out [he said to Davis], the theory, 
the methodology and the conclusions drawn are flawed. I’d go one step further and state that this 
is more likely a failure of PLOS ONE’s peer review process to do even the primitive job to which it 
aspires.” 


Alerted to the criticism Fanelli posted several responses on The Scholarly Kitchen. Eventually giving 
up in frustration he said: “[P]lease, if you really have nothing better to do than try to demolish my 
work, at least read both papers properly before.”?” 


“Welcome to the future of science, at least according to PLOS ONE,” snapped back Crotty. “What 


>,» 


you view as ‘demolishing’, PLOS ONE promotes as ‘post-publication review’. 


Crotty added: “You seem surprisingly uninformed as to the process PLoS encourages. Articles are 
reviewed solely for methodological accuracy — did the author do what the author says he did, no 
conclusions are made regarding the significance of the research. That’s meant to be determined 





°° PLoS ONE’s “Guidelines for Authors” require that conclusions, “are presented in an appropriate fashion and 
are supported by the data.” 

*7 Fanelli had pointed critics to a second paper of his called “’Positive’ Results Increase Down the Hierarchy of 
the Sciences.” (This paper too was published in PLoS ONE). 
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after publication, in public forums such as this one, by the endless, faceless yobs of the internet, 
regardless of their level of expertise or understanding.” 


It is noteworthy that not a single comment on the paper was posted to the PLoS ONE web site 
itself. As Jason Kelly had anticipated, researchers are not inclined to spend valuable time 
commenting on published papers — unless, apparently, doing so on their own site, a blog, or a 
social networking service. 


This is significant since it draws our attention to what might be viewed as a serious potential flaw 
in PLoS ONE’s review process. As we saw, PLoS ONE asks reviewers to make an easier decision on 
whether a paper should be published — on the assumption that the real evaluation will take place 
after publication. 


So the question arises: If a PLoS ONE paper receives no ratings, and attracts no comments, then 
what is its status as peer-reviewed research? After all, when a paper is published by a traditional 
peer-reviewed journal the public assume it to have been checked and certified as sound research 
by qualified specialists, who in the process will give some guidance on its importance. If no 
commentary is ever attached to a PLoS ONE paper, then by the journal’s own criteria is not the 
review process incomplete? 


True, the Fanelli paper did attract comments, but these took place not on the PLoS ONE site but on 
a blog. Moreover, many might feel that the tone and content of the comments was insufficiently 
objective, or scientific, to justify their being described as post-publication review — Crotty’s claims 
about the “faceless yobs of the internet” notwithstanding. 


It seems pertinent therefore to ask how many PLoS ONE papers do attract comments. Mulling this 
question over on The Scholarly Kitchen last April Kent Anderson said: “PLoS ONE believes that its 
‘innovative’ post-publication peer-review approach alleviates some of the obligations of publishing a 
scholarly journal. This innovation is essentially a form of commenting. But as a just-released report 





on peer-review’s role in academia states in discussing PLoS ONE: 


While submitted papers undergo a form of internal pre-publication peer-review, all 
‘technically sound’ papers are published. (A scan of articles suggests that reader comments 
are, in fact, rare.)” 


One of the questions | emailed to PLoS ONE therefore was, “Would it be accurate to say that it is 
generally only those papers that gain media attention that attract comments. What's the proportion 
of articles that get more than 1 comment? How many articles have 0 comments?” 


Unfortunately my question was not answered. 


Last December, however, PLoS did release some figures that offer insight into this. These suggest 
that sceptics are correct: post-publication reviewing is a rarity. Writing on The Scholarly Kitchen 
David Crotty commented: “[Q]uick analysis of the data shows a total of 23,934 articles in the set.*® 
Of these, a mere 1,291 (5.3%) received a ‘rating’ (a user assigned assessment of the article’s value). 
1,042 of those had only one rating, and 208 had two ratings.” 


He added, “As many science journals send articles out for review to three reviewers, that means 
that only 0.17% of PLoS articles received that same level of post-publication scrutiny as far as star 
ratings go. The numbers are similar for articles that received comments (2,606 total papers with 
comment threads, with 1,803 seeing only one comment thread and 512 two comment threads 
yielding 1.2%) and notes (745 papers with notes, 536 with one note, 116 with two yielding 0.38%).” 


Crotty concluded: “Clearly, the research community has yet to embrace post-publication peer- 
review. If traditional methods are indeed in crisis and this is the immediate solution, the result 





** Crotty’s figures will be skewed since the data released cover all PLoS journals. Nevertheless, Crotty’s 
conclusion appears reasonable. 
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then is a literature that is almost entirely unreviewed. I’m not sure that many researchers would 
see that as an improvement.” 


Flooding the media 


PLoS ONE’s habit of using its papers as a tool to promote both itself and OA have tended to fuel 
criticism of its review process. As we said, it also encourages its authors to engage in self- 
promotion too. This inevitably leads to concern about objectivity, and about the quality of the 
underlying science — particularly when exaggerated claims are made. 


PLoS ONE’s guidelines for authors who want to write their own press releases include the following 
advice: “The press often enjoys slightly quirky studies (from flying fungi to grunting worms), 
paleontology (including the Nigersaurus) and those on topical issues, such as climate change 


and biodiversity.” 





Members of the public might be inclined to ask, “Is this the purpose of science: titillating and 
amusing laypeople, and/or feeding their fears and neuroses? Is this indicative of the degree of 
respect that the scientific community has for nonscientists?” 


Certainly some have been surprised at the extent to which PLoS ONE will go to help authors hype 
their papers. Last year, for instance, PLoS ONE published a paper reporting on the discovery of the 
Darwinius masillae fossil (dubbed “Ida”). The authors — who were already in discussions regarding 
the paper with a production company (Atlantic Productions), several television channels (History 
Channel, BBC1, ZDF, NRK) and a book publisher (Little, Brown and Co.) — began hyping the paper 
before it had even been published. 





This led to a flood of wildly overstated stories in the media, including one (in The Daily Mail) 
claiming that the fossil was “a vital ‘missing link’ in human evolution”, an assertion apparently 
encouraged by one of the authors (Philip Gingerich) — who told the press that the paper, “is going 
to advance our knowledge of evolution.” 





While it was not PLoS ONE that made these claims, the publisher showed itself more than happy to 
exploit the media frenzy set in train by the paper’s authors. On the day of publication (May 19" 
2009), for instance, it posted a message on its blog saying that in order to assist the authors in their 
publicity campaign it had published the article ahead of time. 





The post explained, “We published the paper at 10.30 a.m. Eastern Time today, rather than our 
usual publishing time of 8 p.m. ET, because at 10.30 this morning, the fossil was unveiled at 

the American Museum of Natural History (AMNH) in New York, to an audience which included Ellen 
Futter, the President of the AMNH and Michael Bloomberg, the New York City Mayor.” 





The post continued: “The findings have already made and continue to make a significant impact in 
the media (we will post a summary of the coverage later in the week); you can read the freely 
available PLoS ONE article, in full, online on our website (we encourage everyone who writes a 
news story or blog post about the study to link to this URL). A special documentary film, The Link, 
will be screened on US television on May 25"" and in the UK on May 26" and a major book (also 
called The Link) is published on May 20°.” 


With language like this one wonders where a blog post merges into, or itself becomes, a press 
release. (PLoS ONE did itself also put out a press release about the paper, but this no longer 
appears to be available). 


As soon as the paper was published other researchers began to pooh-pooh the exaggerated claims 
that had been made, and to deplore the way the paper had been hyped. “Ida is undoubtedly a 
spectacular fossil,” wrote science writer Brian Switek in The Times. “A nearly complete fossil 
primate, with a body outline and stomach contents, she is the sort of discovery palaeontologists 
dream about. It may come as a surprise, then, that Ida does not change everything we thought we 
knew about human evolution.” 
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PLoS ONE was criticised both for fanning the flames and, more significantly, for withholding the 
paper from journalists who had attended a press conference given by another of the authors —Jgrn_ 
H. Hurum. Without access to the paper, journalists complained, they could not obtain any 
independent scientific views on it, leaving them little option but to recycle the hype, or not report 
at all. 


Writing on his blog Carl Zimmer commented: “[I]t appears that both PLoS and Atlantic Productions 
did not give journalists any time to consult with outside experts before launching a major press 
conference with a huge blitz of media attention. In other words, science writers who were trying to 
do their job well and responsibly were actively hindered. Those who declared ridiculous things, 
such as claiming that human origins were now solved once and for all, were not.” 





Responding on Zimmer’s blog, Binfield insisted that PLoS ONE had done everything it could to make 
the paper available as soon as possible: “Once the paper was accepted we made a strenuous effort 
to publish the article in time for the Press Conference which was happening on May 19" — only a 
week later. We were not involved in the Press Conference, but felt it was clearly in the public 
interest to have the article publicly available in time for that conference. Our production team 
managed to get the article published more than two weeks quicker than normal, so that it would be 
ready for the 19". However, it was only on the afternoon of the 18°" May that we knew the paper 
would definitely be available in time and until that point, no final copy of the paper was 
available.” 


He added, “We do regularly help PLoS ONE authors with the distribution of press releases under an 
embargo, as do many other journals, but when we do this we only ever issue that information on a 
date that is acceptable to the authors. The authors of this paper requested that we did not issue a 
press release, or reveal any other information about this paper, until 10.30 EST on the 19" May (the 
time of the press conference). We respected their wishes, and at the time of publication also 
issued our own press release about this article.” 


Zimmer, however, remained sceptical: “I’m curious what scientists, journal editors, reporters, and 
other readers think of Binfield’s response. Other prominent journals don’t leave these matters up 
to the scientists (or their television producer pals). They inform the scientists of which issue a 
paper is scheduled for, and they put the paper on a press list a few days earlier.” 


Mark Henderson of The Times subsequently emailed Zimmer to report: “[T]he PLoS paper WAS 
made available under embargo to the press — but only to selected individuals and under very 
unusual restrictions. | was invited to read it by Atlantic Productions on Tuesday morning (I’m 
Science Editor of The Times in London), but | had to go to their offices to read it and wasn’t 
allowed to take a copy away. | also had to sign a non-disclosure agreement, which meant | wasn’t 
able to approach anyone else for comment until the embargo lifted. The Guardian also had advance 
access (they got to see more in advance than we did, and earlier). So — obviously because they 
bought the film rights — did the BBC. But other UK papers (Independent, Telegraph etc.) got 
nothing. This is a very weird (and in my experience unprecedented) way to manage the release of 
published science.” 


PLoS ONE does appear to be a little too keen to court the press. As noted, it also it encourages its 
authors to do the same. And Henderson, Switek and Zimmer are surely right to question the wisdom 
of this. The upshot in this case, Switek concluded, was that: “What could have been a unique 
opportunity to communicate science has quickly developed into a fiasco. Science proceeds through 
discovery and debate, and hypotheses do not become accepted by flooding the media with press 
releases.” 


Nevertheless PLoS remains keen to court the attention of the media. In its progress update referred 
to earlier it cites the Darwinius masillae “fiasco” as evidence of the high media interest PLoS ONE 
research gets. And it added that more than 200 articles published in PLOS ONE in 2009 were 
covered by international media and bloggers. 


Could it be that PLoS is struggling to separate its original advocacy role from its new role as 
publisher? 
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Of course, the controversy over the Darwinius masillae paper was not about the quality of the 
research, or of PLoS ONE’s review process, but the overblown way in which the paper was 
promoted, and the willingness of PLoS ONE to collude in the hoopla — even to the point of changing 
the publication date to fit around press conferences and media events. The problem is that this 
tends to raise doubts about its objectivity and about its priorities; doubts that do not sit well with 
the role of a non-profit science publisher. 


It may be that this has become a sensitive issue for PLoS ONE: It is noteworthy, for instance, that 
the only question Binfield directly addressed from the list | emailed was one relating to the hyping 
of papers. (See page 22) 


Hurray for Science! 


Is there a causal relationship between hype and allegations of low quality? The controversy over the 
Darwinius masillae paper would suggest not. But the two things do seem to have a habit of going 
hand in hand. They coincided, for instance, in the controversy over the wind setdown paper that 
was drawn to my attention last August. 


Entitled Dynamics of Wind Setdown at Suez and the Eastern Nile Delta the paper was authored by 
two researchers at the National Center for Atmospheric Research (NCAR/UCAR) at the University of 


Colorado at Boulder (CU). The lead author Carl Drews is a software engineer at NCAR/UCAR; the 
second author, Weiging Han, is an associate professor in the Department of Atmospheric and 
Oceanic Sciences (and Drews’ supervisor). 








What exactly was the issue? In making a scientific case for a natural process referred to as “wind 
setdown”, the paper’s authors used as an example the 3,000 year-old biblical story of the parting of 
the Red Sea to allow Moses to lead the Israelites out of Egypt. 


In a short space of time other scientists, including two of PLoS ONE’s own academic editors, began 
posting critical remarks about the paper. One of those editors — Bjérn Brembs from Berlin’s Freie 
Universitat — made the following comment: “[T]his paper presents a reasonable scientific 
explanation for a documented event that some people for some time took for a religious miracle. 
Hurray for science!” 


The problem, he added, is that, “there is scant, if any, evidence that the documented event ever 
took place, as there appears to be little archaeological evidence of any Jewish habitation in Egypt, 
nor, hence, any exodus. Clearly, scientific phenomena cannot be defined merely in some 
bestseller, or we'd be reviewing heaps of vampire and alien studies; which, in this case, leaves us 
with a paper that explains a phenomenon that most likely never existed.” 


Another researcher called Steven Smith agreed: “Because there was never an Exodus, an attempted 
physical proof of the parting of the Red Sea is beyond silly. The archaeological evidence is 
conclusive: The Jews were never in Egypt, there was never an Exodus, and no Jewish conquest of 
the land of Israel.” 


What seemed to especially annoy critics was that while Moses and the Exodus are mentioned only in 
the abstract (just once and in passing) Drews appeared keen to imply that the paper had provided 
some sort of scientific proof that a mythical event had indeed taken place — notably by means of a 
press release and a video — both of which focused almost exclusively on the biblical story (In fact, 
the paper discussed three different sites). 





As the press release put it, “The biblical account of the parting of the Red Sea has inspired and 
mystified people for millennia. A new computer modelling study by researchers at the National 
Center for Atmospheric Research (NCAR) and the University of Colorado at Boulder (CU) shows how 
the movement of wind as described in the book of Exodus could have parted the waters.” 


As with the publicity surrounding the Darwinius masillae paper, a flurry of breathless and 
exaggerated media stories followed, with coverage by the BBC, Wired, The Los Angeles Times and 
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CNN, amongst others. AFP was sufficiently excited by the press release that it headlined its report 
“Moses had help parting the Red Sea: study”. Drews appears to have encouraged such a conclusion, 
and is quoted saying: “So now there's a scientific basis for a 3,000-year-old story that we've seen 
movies of and read books, and that's really exciting.” 





This was hype. But did the incident reflect badly on PLoS ONE’s peer review process? 


Critics thought it did, concluding that the reviewers had allowed the authors to practice a sleight of 
hand. Commenting on the single reference to the biblical story, another PLoS ONE academic editor 
— Ramy Aziz of Cairo University — said: “I find this statement irrelevant to the scientific topic and 
also culturally biased.” 





Aziz added, “Normally, the Abstract should summarise the paper and not advance a new (here 
pivotal) idea that is not explicitly mentioned in the paper.” 


Responding to his critics Drews commented: “The historicity of the Exodus is outside the scope of 
this paper, which deals with the dynamics of wind setdown at several coastal sites, modern and 
ancient. Since this is a scientific paper in a scientific journal, we have omitted references to what 
a deity did or did not do.” 


In short, Drews’ argument was that his paper made no judgement about the truth of the biblical 
story, or the likelihood of divine intervention, but simply used the mythical event as an example of 
a natural phenomenon. As he put it to one cynic, “We [only] addressed Egyptian history in the 
context of reconstructing the likely topography of the eastern Nile delta circa 1250 BC.” 


The unanswered question of course was, why build a paper around a biblical story in the first place? 
Drews’ critics pointed out that he runs a web site promoting theistic evolution. Amongst other 
things, the web site mentions the Red Sea crossing; and the strap line on the home page reads: 


“faith and science are compatible’”.°” 


Again, it was not PLoS ONE that had put out the press release. Nevertheless, it does encourage 
authors to market their own papers. And as Farley pointed out in connection with the HIV paper, if 
a paper is published in a peer-reviewed journal the views expressed in any associated press release 
inevitably acquire “a public perception of validity.” 


In fact, the wind setdown press release was the work of NCAR/UCAR. | asked the head of media 
relations David Hosansky whether he felt that there had been any sleight of hand in the way the 
paper had been presented to the media. “We are comfortable with the press release,” Hosansky 
replied. “Neither the study nor the press release attempted to make conclusive statements on the 
historicity of the Exodus which, as Carl Drews has noted, is outside the scope of the paper.” Had 
the aim been perhaps to find the oceanic science equivalent of grunting worms and flying fungi? 


So was there a failure on the part of the reviewers? 


Brembs believes so. “I think two mistakes were made,” he emailed me last year. “For one, there 
was no proper reference to the particular mythology the authors referred to. Second, there was no 
reference to alert the reader that the mythology in question lacks empirical support. Both mistakes 
should have been caught by the reviewers, or by the academic editor.” 


One way of judging how effectively a paper has been reviewed would be to examine the reviewers’ 
reports. The PLoS ONE Guidelines for Authors states, “upon publication PLOS ONE routinely posts 
the referees’ reports as comments accompanying the online version of papers.” And it adds, “By 
posting the referees comments, with their permission, PLoS ONE hopes to make the review process 
more transparent as well as stimulating informed debate of the published papers.” 





*» This information was in fact listed in the competing interests section of the paper. 
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In the case of the wind setdown paper, unfortunately, neither of the reviewers’ reports for the 
wind setdown paper were published on the web site. But as Neylon flagged in 2008, very few are 
now posted anyway. 


| nevertheless asked to see the reports. Binfield later replied: “[W]Je did ask both reviewers if they 
would share their reports with you. One replied and expressed that he did not want to share his 
report with you, and the other has not yet replied.” 


| heard nothing further. 


Whether there was a causal relationship between the controversial reviewing of the wind setdown 
paper and the way it was hyped is a matter for speculation. But it is surely not advisable to put out 
press releases with exaggerated claims, or claims that do not accurately reflect the contents and 
conclusions of a paper, as doing so (rightly or wrongly) tends to raise doubts about the quality of 
the underlying research, and therefore of the peer review process it has been subjected to — 
regardless of how much the release might titillate the media. 


Crucial role in expanding open access 


As indicated, it was as a result of the controversy surrounding the wind setdown paper that | 
contacted PLoS ONE executive editor Damian Pattinson. 


One of the questions | sent was this: "PLoS ONE appears to be very keen to spread news about its 
research (not least via its blog), and in ways — some might argue — that gloss over the 
complexities of the research in question. As a result every now and then exaggerated 
interpretations of data are widely disseminated (e.g. the paper we are discussing and, last year, 
the paper on Darwinius masillae), thereby generating negative publicity for the publisher. Yet 
when this happens PLoS ONE always distances itself from what the media say about the articles. 
Given that PLoS ONE courts publicity, and could be said to be partially responsible for the 
exaggerated claims that sometimes emerge about its articles, is it not guilty of hypocrisy?” 


When | received the email from Binfield declining to answer my questions he at least did address 
this point, sending me the following statement (which he requested | quote in full): 


“As regards the media and blog coverage surrounding this article, it occurred several weeks after it 
was published, and was not the result of PLoS issuing a press release about the article. We issue 
press releases on a very small number of the approximately 150 articles that we publish in PLOS 
ONE each week and we did not issue a press release for this article.” 


He added, “There is much further information about PLoS ONE on the journal web site (data on 
the peer review process are provided and the PLoS ONE editorial criteria are detailed), and readers 
are encouraged to contact us if they have any questions.” 


And he concluded, “We believe that PLoS ONE has a crucial role to play in expanding open access to 
new research findings, and tens of thousands of researchers are supporting this effort as editors, 
peer reviewers and authors.” 


Those familiar with the publisher will recognise in the last sentence PLoS’s organisational idée fixe 
about expanding OA — leading one to wonder whether critics might be right to suggest that the 
primary goal of PLoS is no longer that of “shaping the future of publishing”, but “OA at all costs”. If 
correct, PLoS’ priority has presumably shifted from developing high-quality OA journals, to seeding 
the world with so many freely-available papers that OA becomes the default for publishing research 
papers. 


That is undoubtedly a worthy goal, and it seems likely to succeed. But is there a danger that 
adopting such a narrow strategy in trying to solve a problem as multi-faceted as the scholarly 
communication crisis could cause collateral damage — not just in terms of hampering the public 
understanding of science but, as we shall discuss, undermining the very raison d'étre of scholarly 
communication? Could “OA at all costs” be in danger of doing as much (or more) harm as good? 
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Certainly the researcher who drew my attention to the wind setdown paper saw the row it had 
sparked in a wider context. And one suspects that all the paper’s critics did too. 


Scientific rigour 


It would however be wrong to imply that PLoS ONE did not address the issues raised by the wind 
setdown paper: On 30" September last year Pattinson posted an official comment. But | found the 
comment somewhat ambiguous, which is partly why | emailed over my questions. 


Pattinson began by saying that “the scientific rigour of the work has not been questioned”, and that 
the paper had been assessed by an academic editor plus “two relevant experts’. 


Since scientific rigour is one of the main criteria for acceptance by PLoS ONE, it is unsurprising that 
Pattinson stressed the point. He did however appear to concede that the review process had not 
been adequate. Specifically, he suggested that the paper “could have been framed, interpreted and 
discussed differently.” Nevertheless, he added, PLoS ONE did not feel there were “any grounds to 
retract or formally correct the article.” 


Again we find ourselves wondering exactly what scientific rigour is. And if scientific rigour is the 
litmus test for whether a paper should be published, can we be confident that the review process is 
adequate? 


Let’s remind ourselves that Varmus’ description of PLoS ONE was “a very large compendium of 
papers that have been vetted for scientific quality, but which will not be confined in terms of their 
likely importance.” One question to ask with any controversial paper published by PLoS ONE, 
therefore, is whether it could be said to have met that standard (assuming we are confident we 
know what scientific quality is). 


Pattinson added: “there are lessons that we can learn from the reaction to this article (and to other 
recent articles that have generated criticism)”. However the implication seemed to be that 
ultimate responsibility lay with the academic community. As he explained, while PLoS ONE staff 
“screen” incoming papers for such things as potential conflicts of interest, “[o]nce the articles are 
assigned to academic editors, we rarely intervene in the peer-review process but we provide as 
much support to the editors as we can, and we will redouble our efforts to do this in future, so that 
we can ensure that all articles, especially controversial ones, are presented as clearly and 
accurately as possible.” 


Stressing this point, Pattinson said that PLoS ONE “is run by the broader academic community, using 
a distributed editorial model whereby individual academic editors take responsibility for specific 
editorial decisions.” 


Cynics might have concluded that Pattinson was trying to pass the buck, implying that if any 
criticism is due it should be directed at the reviewers, and the academic editor, not at the 
publisher. But that would be to ignore the fact that PLoS ONE defines what criteria must be applied 
when reviewing papers, not the reviewers, and not the academic editor. In other words, it is PLoS 
ONE that tells reviewers to ask simpler questions and to make easier decisions. And it is PLoS ONE 
that insists on a fast turnaround (inevitably putting pressure on reviews to also deliver fast, rather 
than necessarily thorough, decisions). 


Pattinson concluded by pointing out that PLoS ONE expected to publish around 7,500 articles by 
the end of 2010, becoming “the largest peer-reviewed publication in the world.” For that reason, he 
said, “It is inevitable that PLoS ONE will publish articles that occasionally touch on controversial 
topics”. 


That would seem to imply that the row had been over the topic of the paper, not its quality, or the 
adequacy of the review process it had been subjected to. It may be that critics disliked the topic, 
but their real beef surely was with the way that the paper had been written, what they viewed as a 
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sleight of hand on the part of the authors, and the failure of PLoS ONE’s review process to see that 
this was corrected before publication. 


Production values 


As we see PLoS ONE’s appear to have a habit of attracting criticism. But is it simply a by-product of 
publishing so many papers (as Pattinson implied) or is PLOS ONE’s review process inadequate — as 
critics imply? The papers discussed above certainly raise questions, both about PLoS ONE’s review 
process, and the way in which the publisher communicates science — not least by encouraging 
authors to engage in self-promotion. 


In terms of adequacy, there are a number of more specific issues one could look at too. For 
instance, PLoS ONE does not copyedit the papers it publishes, only the abstracts. But it would 
appear that even this minimal service is not always provided. That this can be problematic became 
apparent to me in November 2008, when a researcher drew my attention to an article whose 
abstract was barely comprehensible. 


The paper was called “Incompetence of Neutrophils to Invasive Group A streptococcus Is Attributed 
to Induction of Plural Virulence Factors by Dysfunction of a Regulator”. The abstract was riddled 
with spelling errors, malapropisms, and grammatical errors, and included the following incoherent 
sentence: “In addition, rare outbreaks of invasive infections and their distinctive pathology in which 
infectious foci without neutrophil infiltration hypothesized us invasive GAS could evade host 
defense, especially neutrophil functions.” 


When | asked Binfield to comment he agreed that there was a problem, replying: “Having looked at 
this example, clearly the language here wasn’t up to scratch. | queried our production department 
and it seems that this abstract slipped through the cracks and was not copyedited properly.” 


He added, “I am going to be following up with the Academic Editor who handled this paper (Dominik 
Hartl — as named on the paper) to see what his opinion is, and whether we should make a formal 
correction to PubMed as a result.” 


When | looked at the abstract again last October | noticed that the errors were still there. This time 
| contacted Hartl, and | asked him what the outcome of his conversation with Binfield had been. 
Hartl, however, seemed unaware of the issue, and asked: “What do you feel is in its current version 
incorrect in this abstract?” 


| suggested he start by reading the sentence cited above, and go on from there. Later | received a 
follow-up email from Hartl. He wrote: “That definitively needs to be corrected, | agree.” 


He added, “Normally, PLoS ONE proof-read. Editors should have corrected this prior to 
submission..... . 


Once again, however, my enquiry appeared to fall into a black hole. At the time of writing the 
abstract remains in its uncorrected form, both on the PLoS ONE site and in the PubMed and PubMed 
Central records two years after | first contacted Binfield, and over four months after | contacted 
Hartl. 








Perhaps this is unsurprising: When | contacted Binfield he seemed to be signalling that he does not 
take such matters very seriously. Explaining PLoS ONE’s “production values”, he said: “Speaking for 
PLoS ONE we do not copyedit content (other than a very light clean up). We do a light (but real) 
copyedit on the abstract; and at time of submission one of our (many) Quality Control checks is on 
the quality of the English. However, as a general rule, if the language is intelligible, and passes QC 
and passes peer review etc., then it will be published as is.””” 





“It is worth noting that the PLoS ONE guidelines state quite clearly, “Perfect, stylish English is not essential 
but the language must be clear and unambiguous. If the language of a paper is poor, Academic Editors should 
recommend that authors seek independent editorial help before submission of a revision.” Elsewhere it says, 
articles must be “presented in an intelligible fashion”, and “written in standard English.” 
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Maybe Binfield’s views are general amongst scientists today, and so unremarkable.*' However, 
members of the public — who fund most of the research published in PLoS ONE, and who fund 
PubMed and PubMed Central — might wonder why it is thought acceptable for the abstracts of 
scientific papers to verge on the incomprehensible. They might also wonder how, if this is the end 
product, PLoS ONE can justify charging $1,350 to publish a paper. 


Since the taxpayer additionally pays for researchers’ time when they do the reviewing (via their 
salary), the public might be forgiven for concluding that it is getting pretty poor value for its tax 
dollars. Should not charging a fee of $1,350 at least require the publisher to run a spellchecker over 
a 267-word abstract? Should not professional scientists be expected to describe their research in an 
articulate and coherent manner when putting it into the public domain? 


And if Butler and Davis are right to claim that PLoS ONE is overcharging for its services in order to 
subsidise its sister journals is it not unfair on PLoS ONE authors (and their funders)? Why should 
scientists who have failed to get their papers published in a prestigious journal be expected to 
subsidise authors who have succeeded in doing so?” 


Pointless? 


Back to the question of whether PLoS ONE’s review process might be deemed inadequate or 
problematic. As we've said, some believe that it is — not just critics of PLoS like Davis, Crotty and 
Anderson, but PLoS ONE academic editors too. 


Indeed, at least one PLoS ONE author has his doubts too. Last year a researcher whose first paper 
had just been accepted by the journal copied me into an email he sent to his co-authors. He wrote: 
“My opinion of PLoS ONE (which | usually praise) has gone down considerably given this very 
superficial and purely cosmetic refereeing, which gives no evidence of having understood the paper 
or even having read it critically rather than just self-interestedly.” 


If PLoS ONE’s own customers believe that the service it provides is inadequate then should we not 
all be concerned? 


But if there is a problem what exactly is it? 


There are a number of issues we could explore further — not least the implications of asking 
reviewers to make easier decisions, and the ambiguity of some of the instructions given to referees 
— but one of the most frequently cited concerns relates to PLoS ONE’s insistence that reviewers 
take no account of significance when evaluating papers. 


This is the issue that critics like Davis and Anderson most frequently raise — not (just) because it 
means abandoning the filtering role that journals have traditionally played, but because they 
believe that attempting to review a paper purely on the grounds of technical soundness (the 
consequence of ignoring significance) can lead to official certification of pointless research. As 
Davis argued last October, it means reviewers are required to accept papers that have little or no 
inherent value. 


If correct, this would suggest that PLoS ONE is in danger of encouraging researchers to waste their 
own time, and thus the time of their reviewers, and any readers. One might also wonder why the 
taxpayer is expected to pay $1,350 to publish a paper that has little or no worth. 


The question to ask, suggested Davis, is: “Can a scientific paper be methodologically sound, but just 
not report any significant or meaningful results?” He added, “What constitutes whether a paper that 
is ‘technically sound’ is much more nuanced and much less clear than it appears. In fact, you will 
not find discussions of what makes a sound methodology in any methods textbook.” 


“' This discussion, however, suggests otherwise. 
“ One assumes that, all things being equal, any researcher would prefer to be published in PLoS Medicine than 
in PLoS ONE 
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In other words, what is the value of a paper — even if it is technically sound — if its publication is 
pointless?” 


To make his point about methodology Davis asked a number of colleagues from different disciplines 
to define what they understood by sound methodology. The replies he received varied 
considerably, but were generally too vague to be useful (e.g. “appropriate methodology is what 
your community accepts as appropriate methodology. ”) 


So we are left wondering whether scientific rigour and technical soundness are adequate tests. It 
could be that PLoS ONE’s approach would work with a journal addressing a small sub-discipline. But 
in aiming to cover “all disciplines within science and medicine” can it really hope to provide a 
uniform approach to peer review? How can reviewers from different disciplines know what PLoS 
ONE means when it asks for scientific rigour and technical soundness if there is no science-wide 
consensus on sound methodology? And in the absence of clear definitions, how can we judge 
whether PLoS ONE is measuring up to its own standards? 


And where a journal has an editorial board as large as the one now deployed by PLoS ONE its review 
process must surely start to become unmanageable, and at some point impossible to police. As we 
saw, PLoS ONE has around 1,000 academic editors. This figure, | am told, is set to double in the 
near future. 


One of the questions | asked PLoS ONE was: “Would you agree that when an editorial board gets as 
large as PLOS ONE’s it is very difficult to have unified standards? Academic editors are inevitably 
like judges who will always make different rulings. If the number of academic editors keeps 
increasing there will surely come a point where there will be no standards at all. What one editor 
rejects will be perfectly acceptable to another editor. This is surely demonstrated in the case of 
the wind setdown paper: Two PLOS ONE academic editors have signalled that they would not have 
accepted it — in its current form at least — and yet it was accepted by another editor?” 


| received no answer to this question. 


| also asked this question: “I understand that PLOS ONE currently has 1,000 academic editors, and 
this is set to increase to 2,000. Is there not a danger of creating ‘families’ within the editorial 
board? An editor could, presumably, invite a bunch of friends who are not affiliated with him 
directly to submit papers and guarantee that if they submit a paper s/he will guarantee it will go to 
friendly reviewers. After all with numbers like that nearly every author will have a friend in the 
board. Is that not a recipe for potential corruption?” 


Again | received no answer to my question. 


And as | had been told that PLOS ONE academic editors can, if the wish, simply waive papers 
through without sending them out for review, | also asked PLOS ONE the following question: “I 
understand that academic editors are able to accept a manuscript right away without seeking a 
second opinion. Is that true? If so, is that not also likely to encourage corruption? Do you have any 
figures on the number of papers that are immediately accepted by academic editors?” 


Alas, no answer to this was forthcoming either.“ 


Allin all, one is inclined to conclude that PLoS ONE’s review process must be somewhat arbitrary 
and, let’s face it, rather unscientific. 


43 For authors, of course, it will not be pointless, since publication adds another line to their CVs, and so helps in 
terms of tenure and promotion. But the paper may nevertheless be pointless in terms of adding value to 
mankind’s scientific endeavour. 

“* Tn the links that Binfield sent me, however, I did find this statement: “[Academic editors] can conduct the 
peer review themselves, based on their own knowledge and expertise.” Another link led to a note_ saying 8% of 
the articles in PLoS ONE are reviewed by the editors themselves. 
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Journal form at risk? 


Where does this leave us? We’ve suggested that PLoS ONE raises two interconnected issues for 
scholarly communication — one relates to quality, and the other to costs. And in considering these 
issues our attention has also been drawn to the problems inherent in promoting scholarly papers as 
though they were products. Doing so seems inevitably to lead to exaggerated claims, and hinders 
rather than aids the public understanding of science. 


In considering the topic of quality the perennial issue raised by OA critics is that in an author-side 
fees model the primary customer becomes the author rather than the reader. This, they say, has 
implications, not least that it inevitably exerts a downward pressure on quality, since publishers 
face much greater pressure to accept papers than they do in a subscription model. 


In a memorandum submitted to the UK Science & Technology Select Committee in 2004 Elsevier 
described the danger this way: “By introducing an author-pays model, Open Access risks 
undermining public trust in the integrity and quality of scientific publications that has been 
established over hundreds of years. The subscription model, in which the users pay (and institutions 
like libraries that serve them), ensures high quality, independent peer review and prevents 
commercial interests from influencing decisions to publish. This critical control measure would be 
removed in a system where the author — or indeed his/her sponsoring institution — pays. Because 
the number of articles published will drive revenues, Open Access publishers will continually be 
under pressure to increase output, potentially at the expense of quality.” 


In other words, because in an OA model authors buy a service from the publisher, the author 
becomes the ultimate customer. Since authors need to have their papers published in order to 
advance their careers, and OA publishers know that every sale will increase their income, the 
pressure to prioritise income and careers over quality becomes too great to ensure scientific 
objectivity.” 


What Elsevier did not predict is that OA would spark a “gold rush” of new author-side fees 
publishers offering services more akin to vanity publishing than scholarly communication. That, 
however, appears to have happened. In a 2010 review of some of these companies US librarian 
Jeffrey Beall dubbed them “predatory publishers”. They are predatory, he argued, because their 
mission “is not to promote, preserve, and make available scholarship”, but to “exploit the author- 
pays, Open-Access model for their own profit.” 





Elsevier likewise did not predict (although it was perhaps implied in its memorandum) that author- 
side fees publishers might deliberately introduce a “simpler” or “lighter” review process as PLoS ONE 
has done. 


We should not doubt that PLoS’ primary motivation is to further the cause of OA, not (as is 
doubtless the case with predatory publishers) to make a financial killing. But if, as critics claim, 
PLoS ONE’s APC has been deliberately over-priced in order to subsidise PLoS’ flagship journals we 
should be concerned — not least because it has implications for scholarly communication at large. 


What are these implications? First, there is a danger that quality will be sacrificed on the altar of 
profit. For that reason, some have concluded there is little to choose between PLoS ONE and Beall’s 
predatory publishers. This is implied, for instance, in an April 2010 post on The Scholarly Kitchen by 
Kent Anderson, who wrote: “Author-pays journals make money when authors publish. Some, 

like PLoS ONE and Bentham Science, work to get as many papers as possible published in order to 
bolster their bottom lines.” 





* OA publishers deny there is any commercial pressure on the quality of the papers they publish, arguing that 
publishing decisions are always taken by academic peer reviewers and editors, independently of the publisher. 
Editors of OA journals concur. Critics, however, are sceptical about such claims, particularly in the wake of a 
number of incidents in which OA journals have accepted flawed, or even nonsense, papers for publication. In 
one case, the editor resigned after a computer-generated paper was accepted, claiming that he had not even been 
aware the paper had been submitted, let alone sanctioned its publication. 
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In the case of PLoS ONE, added Anderson, the revenues generated are far from insignificant. “As of 
this writing, the PLoS ONE site lists 122 articles published in the past week. At $1,350 per article, 
that’s ~$165,000 last week in publication charges, equating to ~$8.5 million per year. This doesn’t 
factor in the institutional charges that can offset some of these.”” 





And PLoS ONE submissions continue to grow. As we saw, by the end of 2010 the journal had 
published 15,238 papers since launch. At the time of writing this had grown to 17,099. 


Let’s say it again: there is no reason to assume that PLoS ONE’s motivations are in any way dubious, 
or that it has ever done anything unprofessional (and let’s assume for a minute it is not 
overcharging PLoS ONE authors). Even so, in making author-side fees publishing and “light” 
reviewing respectable PLoS has helped drive down the quality of published research, if only by 
enabling the so-called predatory publishers to prey on the research community.” 


Anderson believes that high-volume publishers like PLoS ONE (along with its less reputable clones) 
pose a second, greater, danger. In lowering the quality of published research, he argues, they 
threaten to destroy the very notion of the scholarly journal. “Noise, chaff, and pollution in science 
should be controlled upstream by scholarly publishers,” he wrote. “Journals are well-positioned to 
make a difference here, but the way PLOS ONE and similar publishers are bulk-processing 
manuscripts into journal dress could possibly devalue the journal form at its roots.” 


He added: “If being published in a journal no longer immediately carries the imprimatur of having 
cleared a high bar of scrutiny, then the form itself is at risk. Journals could become merely 
directories of research reports. And publishers who are truly setting standards should take notice of 
the risk the drift toward directories poses.” 


Davis makes the same point. However, he believes that the benefits of moving towards “directory 
publishing” are sufficiently compelling that the PLoS ONE model will prove highly contagious.” 
Reporting on the launch of BMJ Open last year, for instance, Davis pointed out that even traditional 
subscription publishers are succumbing to the downward drift. 


Citing an advertisement for a managing editor for the new BMJ Open journal, Davis commented: 
“BMJ clearly sees a market for their new open access journal. But that’s it — a market, a business 
opportunity. The ad is devoid of any lofty goals and aspirations for how this journal is going to 
improve medicine. BMJ Open represents a business decision to recapture manuscripts (and article 
processing fees) that would have been lost to other publishers. It is a bulk publishing model, not 
unlike PLoS ONE.” 


Davis’ point is that subscription publishers like BMJ are coming to realise that by introducing new 
OA journals operating light peer review they can offer what he calls “cascading peer review” 
services. And doing so would allow them to earn a fee from papers that have been rejected by their 
more fussy subscription journals. Once a paper is rejected, the model assumes, the publisher could 
invite the authors to pay to have it published in a lower-quality (i.e. less-choosy) OA journal 
instead. This would allow the publisher to earn income on papers that would otherwise end up 
being published by an exclusively OA publisher like PLoS ONE. 





“© This refers to the various membership schemes OA publishers have introduced to help research institutions 
defray APC costs for their scientists e.g. here. 

“7 We should note that the handful of journals named in Beall’s article represents just the tip of an iceberg. 
These new journals come from companies whose ownership, status and peer review quality are usually obscure 
or unknown. Yet researchers publish in them — such are the “publish or perish” pressures on them today. 
Consider, for instance, that on 24% February I received, out of the blue, a manuscript from one of these 
predatory publishers inviting me to review it. I have never had any dealings with the publisher, and am not an 
academic but a journalist. And for this the author is expected to pay a $550 handling fee. 

“8 As we shall see, Davis has been proved right: publishers are now rushing to copy PLoS ONE. 
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If this were to become common an inevitable side effect would be that (for all practical purposes) 
every paper authored would be published, regardless of its quality. It would also mean that OA 
would, in the eyes of researchers at least, become synonymous with low quality. 


“For many authors resigned to the fact that they have been rejected by a top-tier journal, a 
second-tier specialist journal will do just fine,” said Davis. “[A]nd barring those, a general archival 
journal may be better than no publication at all.” 


Not new 


But hang on a minute. Is this not a bit simplistic? Let’s not forget that the critics of PLoS ONE we 
have been citing — Butler, Anderson, Davis, Crotty et al. — are not objective observers. They are 
advocates of (and have a vested interest in) subscription publishing, and in the traditional notion of 
peer review. This does not make their criticism invalid, and it does not mean that they are wrong, 
but it does overlook an important fact: Many of the weaknesses and faults that they draw attention 
to in OA journals are prefigured in subscription journals.” 


First, cascading peer review has been a characteristic of scholarly publishing for many years, 
although papers have tended to cascade from publisher to publisher rather than from journal to 
journal within a single publisher’s portfolio. OA advocate Stevan Harnad uses the term hierarchy of 
journals to describe this phenomenon, but cascading peer review would be equally descriptive. 
Both terms denote a spectrum of quality, and both imply that practically every paper is sooner or 
later deemed publishable. As Harnad put it in 2007, “[E]verything does get published, eventually, 
but there is a hierarchy of journals, with a corresponding hierarchy of peer-review quality 
standards.” 


In practice, this means that when trying to get a paper published researchers will first submit it to 
a top journal. If that journal rejects it they re-submit it to a less prestigious journal, and so on 
down the ladder of quality until their work is accepted. 


What’s new is that in their restless pursuit of profits publishers are assembling ever larger portfolios 
of journals. This opens up the possibility of their being able to offer their own quality hierarchies, 
eventually becoming one-stop publishing shops for researchers. So any scientist submitting a paper 
to a big publisher could be confident that it would be accepted, although perhaps not in their 
journal of choice. 


In a subscription environment the ability to offer cascading peer review is appealing, but not 
compelling. In an OA environment, however, the logic becomes irresistible, especially for 
subscription publishers. Confronted by growing competition from OA publishers they have now 
learned to love OA — not least because they have come to realise that by offering Hybrid OA’ they 
can now earn both subscription revenues and APC fees from the same journal. Cascading peer 
review promises yet another attractive new revenue stream for them.” Importantly, it could 
neutralise the threat posed by the new generation of OA publishers, particularly those like PLoS 
ONE whose encyclopaedic approach allows them to offer the equivalent of hundreds of journals as 
a large database service. 





* As we also saw, some of PLoS ONE’s own editors and authors have concerns about the journal too. 

°° Hybrid OA is a publishing model in which a subscription publisher offers researchers the opportunity to pay 
to have their article made freely available on the Web. While most of the contents of the journal will remain 
behind the subscription paywall, the researcher is able to ensure that his or her individual paper is OA. The first 
Hybrid OA initiative (Open Choice) was launched by Springer in 2004. 

°! With PLoS ONE as a model publishers can reduce their costs by providing a minimal service, while 
continuing to overcharge the research community for what they provide. Meanwhile Hybrid OA publishers feel 
able to charge over twice as much — $3,000 in the case of Open Choice. 

** Tt is interesting to note that OA publisher Hindawi was recently driven to publish a press release boasting of 
the total number of papers submitted to it in December 2010 (as though its more than 200 journals were one big 
box like PLoS ONE). 
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Neutralise how? As we said, authors rejected by a prestigious subscription journal could be gently 
shepherded to a lower-quality author-side fees journal and invited to pay to publish. Unlike 
subscription publishers, however, OA publishers don’t generally have any prestigious journals to 
offer authors, so they have no honey pot with which to draw them in. By launching new OA journals 
subscription publishers could gain a potentially significant competitive advantage. 


Second, while critics like Davis and Anderson highlight the problems PLoS ONE faces in trying to 
offer meaningful review standards, the truth is — as we pointed out earlier — there have never 
been any universally agreed methods, standards, or formal processes setting out how peer review 
should be conducted. 


Partly for this reason, no doubt, subscription journals often discover that they too have unwittingly 
published low-quality, wrong, sometimes even fraudulent, research (see, for instance, here, here, 
here, and here). Indeed, even the most prestigious journals in the world are not immune from 
publishing bad or falsified research, and the occasional drivel too.** And the number of incidents 
has been growing in recent years. Peer review scandals, in short, are on a growth curve quite 
regardless of OA.” 


OA aside, therefore, something else is going on. That something is an exponential growth in papers 
produced (since the end of WWII at least), accompanied by an apparent decline in the quality of 
peer review. The latter is doubtless a consequence of the research community’s inability to cope 
with the growth. In the same year that Varmus sat down with Pat Brown in a San Francisco coffee 
shop The New York Times concluded that the number of scientific articles and journals being 
published around the world “has grown so large that it is starting to confuse researchers, 
overwhelm the quality-control systems of science, encourage fraud and distort the dissemination of 
important findings. 


The NYT cited the deputy editor of the Journal of the American Medical Association (JAMA) 
Drummond Rennie, who said: “There seems to be no study too fragmented, no hypothesis too 
trivial, no literature too biased or too egotistical, no design too warped, no methodology too 
bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis 
too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no 
grammar and syntax too offensive for a paper to end up in print." 








Amplified 


In short, peer review was in trouble before PLoS ONE appeared on the scene. The absence of peer- 
review standards, and an inability to keep pace with the constant growth in research papers, has 
caused the system to begin to creak. Rather than being an issue unique to OA journals, therefore, 
quality control is now plaguing scholarly communication at large. 


What is new, however, is that in propagating a new, lighter, review process PLoS ONE has amplified 
the pre-existing problem. Importantly, PLoS ONE has done this in the name of a worthy cause 
(making research freely available on the Web). In short, it has formalised and legitimised a process 
that looks set to accelerate the current decline in the quality of published research. 





°° Tf there are no clearly understood rules, of course, essentially everything is publishable, somewhere, 
somehow. 

** It’s a real case of the more you look, the more you find. In 2005, for instance, Science withdrew a number of 
papers co-authored by South Korean scientist Dr Hwang Woo-suk after it was revealed that he had faked the 
results of key stem cell research. On February 2"! 2010, The Lancet withdrew a 1998 paper co-authored by 
controversial former British surgeon Andrew Wakefield in which a connection was claimed to be made between 
the MMR vaccine and autism. The Lancet noted that elements of the manuscript had been falsified. Last 
December The Scientist published an article headed, “Top Retractions of 2010”. And so the list goes on ... See 
also here for instance and here. 

°° In an attempt to maps this last summer a blog was launched called Retraction Watch. This aims to monitor 
and publicise papers being retracted. 
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In addition, PLoS ONE has demonstrated to other publishers that lowering the bar of scrutiny offers 
compelling financial benefits. And it justifies this on the grounds that papers can only be properly 
reviewed after publication, and that testing for “technical soundness” is the same thing as 
conducting “rigorous peer review” — leading one to suspect that it is promulgating a form of 
doublethink. 


Sooner or later, critics argue, it will become apparent to all that the emperor has no clothes.” At 
that point a system (peer review) that was always little more than a compromise, and primarily 
form over substance, will become sufficiently discredited that the current system of scholarly 
communication will collapse. 





Indeed, the alarm bells are already ringing: In January the UK Science & Technology Select 
Committee announced a new enquiry into peer review. 


Whether systemic collapse would be a good or bad thing depends on one’s point of view. One could 
argue, for instance, that if scholarly communication was viewed simply as a way of making research 
visible to peers, rather than a process in which papers were checked, approved and certified, this 
would not matter. 


But since the current system assumes that published research has been checked and certified — 
when in reality this may not have been done adequately — one might conclude that the public is 
being misled, and it is therefore time for the research community to go back to the drawing board. 


In passing, we should also point out that the habit of promoting research papers as if they were 
products is not unique to PLoS ONE. Last December, for instance, a paper published in Science by a 
group of NASA-based scientists claimed to have discovered an arsenic-based life form (bacterium). 


The paper was the recipient of considerable hype. This included an advance press advisory 
trumpeting that NASA scientists would soon reveal an “astrobiology finding that will impact the 
search for evidence of extra-terrestrial life”, followed a few days later by a press release 
announcing that “NASA-funded astrobiology research has changed the fundamental knowledge 
about what comprises all known life on Earth.” 


In a short space of time, however, the blogosphere had given the paper a big thumbs-down, anda 
wave of criticism spread across the Internet — spearheaded by Rosie Redfield, a microbiology 
professor at the University of British Columbia. The paper, she declared contained “lots of flim- 
flam, but very little reliable information.” 


When | emailed Redfield she added that, in her view, the paper was based on faulty research, was 
inadequately peer reviewed, and represented, “bad judgement by the authors, reviewers and 
editor”. 


Once again, however, PLoS ONE appears to be taking hype to the next level, not least by 
encouraging authors to produce their own press releases. And once again we are tempted to 
conclude that an “OA-at-all-costs” strategy could simply amplify the pre-existing problems of 
scholarly communication. 


Affordability 


But let’s return to the question of costs. As we indicated, this is tightly bound up with quality. By 
demonstrating that it is possible to charge authors to publish (rather than levy a subscription on 
readers to read), and by showing that it is possible to charge $1,350 for a service sufficiently 
minimal that papers are not even copyedited, PLoS ONE has pioneered a model that holds out to 





°° None of this is to imply that where reviewers are committed to providing a fair-minded critique, and have the 
time to do so effectively, and where paper authors have submitted a paper in a genuine attempt to communicate 
valuable scientific results (rather than simply add another notch to their publication record), peer review cannot, 
and does not, provide a valuable service, and an effective filter. Such motivations, and the necessary time, 
however, appear to be increasingly rare in today’s stressful publish-or-perish environment. 
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subscription publishers the promise of being able to migrate to an OA environment without any loss 
of revenue (perhaps to earn even greater revenues), and without any reduction in their high profit- 
margins. 


It might of course seem entirely reasonable for publishers to expect to continue earning a good 
living in an OA environment. But we should note that one of the benefits of OA promised by OA 
publishers was that it would reduce the costs of publishing. We should also remember that 
publishers have long been criticised for making excessive profits from the research community. And 
we should note that the research community is approaching the point where it will no longer be 
able to afford publishers’ asking price.” 


It could therefore be that by taking too narrow an approach to the crisis afflicting scholarly 
communication (“OA at all costs”), PLoS ONE will not only encourage a decline in quality but 
exacerbate rather than ameliorate the financial difficulties confronting the research community. In 
an environment where research output is constantly growing,”® the implications of this could be 
grave, and could on their own trigger systemic collapse. 


In other words, although PLoS ONE is rapidly increasing the quantity of research that is freely 
available on the Web, it is doing nothing to resolve a financial crisis that has had scholarly 
communication in its grip for several decades. 


Known historically as the serials crisis’, in the context of OA it would be more accurate to call it an 
affordability crisis. Rather than address the problem, PLoS ONE has opened up a new seam of gold 
for publishers, and provided them with a migration path that will allow them to lock their current 
profit levels into the burgeoning OA environment, regardless of the savings they make from OA, and 
from electronic distribution. 


What has become apparent is that simply shifting the costs from the end of the publishing process 
to the beginning — from reader to author, from subscription to APC — has not removed any costs 
from the system, but simply reallocated them. Indeed, as we shall argue, there are reasons to 
believe that the author-side fees model may exacerbate the affordability problem. 


To do PLoS justice, when Varmus left the San Francisco coffee shop 13 years ago his focus was not 
on the costs of the scholarly communication system, but on improving access to research. 
Nevertheless, PLoS assumed that replacing subscriptions with APCs would also provide costs 
savings. 


In a written statement to the UK Science and Technology Select Committee in 2004, PLoS said, 
“New technology — most notably the Internet, with its capacity for distribution, storage, and 
retrieval of information in digital form — has brought us to the threshold of an era that scientists, 
philosophers and scholars have hoped for and dreamed about for millennia.” 


That PLoS assumed this would mean lower costs was evident in what Varmus told the Committee 
when giving oral evidence: “There is no doubt that when you create one copy in digital form which 





°’ The affordability problem is composed of several phenomena. First, as research continues to grow 
exponentially so the costs of communicating that research grow. Second, the amount of money research 
institutions can afford to pay to support scholarly communication (subscriptions, APCs, OA memberships etc.) 
continues to fall. Third, publishers have nevertheless (until now at least) continued to earn excessively high 
profits from publishing research. And they can do this because the scholarly journal market is dysfunctional. See 
the analysis of the market, for instance, by anti-trust economist Mark McCabe: 
http://www.infotoday.com/it/dec02/poynder.htm. 

** Recently the CEO of Springer (owner of BMC) estimated growth at between 6% and 7% per annum. 

»* As Wikipedia puts it, “The term serials crisis has become common shorthand to describe the chronic 
subscription cost increases of many scholarly journals. The prices of these institutional or library subscriptions 
have been rising much faster than the Consumer Price Index for several decades, while the funds available to the 
libraries have remained static or have declined in real terms. As a result, academic and research libraries have 
regularly cancelled serial subscriptions to accommodate price increases of the remaining current subscriptions.” 


32IPLOS ONE, Open Access, and the Future of Scholarly 
Publishing 





can be used by everybody, you have a simpler and less expensive system than if you require the 
publisher to carry printed copies; every copy costs an additional fee to the system.” 


The implication was that electronic publishing (which OA assumes) would be significantly cheaper, 
and that publishers would automatically pass the savings on to the research community. It was 
also assumed that new automated editorial and production systems would provide cost savings; 
and it was argued that further savings would be realised as a result of publishers no longer having 
to administer expensive subscription systems etc. 


Asked by the Select Committee how much it costs to publish a paper in an OA environment, PLoS 
said, “We estimate that a typical article processed through our journal management system and 
production process costs between US$1,000-1,200, not including costs for peer review and print”. 


It added: “[F]or our future PLoS community journals (with staffing and publication standards similar 
to most society journals), we estimate that peer review will cost no more than US$200 [£100] per 
article”. 


In other words, PLoS argued that it would be possible to publish a traditional peer-reviewed paper 
for between $1,200 and $1,400. We can assume it would have predicted significantly lowers costs 
for a journal like PLoS ONE that utilised a light review system (PLoS ONE was launched two years 
later) — particularly where it did not even copyedit papers and, by publishing nearly 7,000 papers 
a year, would inevitably benefit from significant economies of scale. 


Today PLoS charges $2,250 to publish in its community journals, and $2,900 to publish in PLoS 
Biology and PLoS Medicine. These costs continue to rise, mirroring the serial price inflation 
characteristic of traditional subscription publishing. As indicated earlier, PLoS ONE has also 
increased the cost of its APC since launch. 


The same inflationary process can be observed with PLoS ONE’s main OA competitor BioMed Central 
(BMC). When BMC launched it charged $525 to publish an article. Today its standard APC is $1,605, 
and authors can pay up to $2,440 to publish in a BMC journal like Genome Biology. 


Meanwhile, the cost of publishing in a hybrid-OA journal ranges from around $3,000 per paper with 
Springer’s Open Choice, to $3,850 with BMJ. Per-paper costs, in other words, are rising inexorably. 
All things being equal, at some point we could expect the price to rise to the figure Nature cited 
when speaking to the UK Science & Technology Select Committee in 2004. 


Author crisis 


The mistake PLoS made, it seems, was to assume that the savings made possible by OA would be 
passed on to customers. 


In doing so it overlooked two important facts. First, the scholarly journal market is not a proper 
market. As such, the mechanisms that operate to control and restrain prices in a true market are 
absent. For this reason, antitrust economist Mark McCabe has described it as a broken market.” 


Second, PLoS has surely discovered that if you don the robes of a publisher you acquire the 
trappings, the tastes, and the associated costs of being a publisher, even if you are inspired by a 
vision of something better. As a consequence, PLoS appears to be as focused on maximising its 
income today as any other publisher. 


° Of course subscription publishers were already making their journals electronic. In doing so, however, they 
were locking in their prices, and forcing libraries to buy large bundles of journals by means of the infamous 
“Big Deal”. 

*! As the launch press release put it, “The software is open source, and will form the first part of an innovative 
and flexible publishing system that will be developed over the next two years and will be available to all groups 
for storing, disseminating, and sharing literature and data.” 

°° For an analysis see this interview with McCabe from 2002 
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This was all the more likely given PLoS’ determination to compete with the world’s most 
prestigious journals. This saw it, for instance, locate its HQ in one of the most expensive cities in 
the world (San Francisco), poach high-salaried staff from journals like The Lancet, Cell and Nature, 
and hire a chief executive from the financial services industry on a salary of $338,000. Most 
bizarrely, it led to PLoS burning money on expensive TV air time to promote itself (to whom it is 
not clear).® 


In short, in order to meet the costs arising from competing with the world’s top journals, PLoS has 
found it necessary to tax the research community at the same high level as traditional publishers. 
And, argue critics, the bulk of this comes from overcharging PLoS ONE authors. 


As indicated earlier, there is a further complication: Even if publishers’ charges were demonstrated 
to be fair and reasonable, the research community would still be unable to afford them in the long- 
term. Grappling with static or falling budgets, research institutions simply cannot meet the rising 
costs of scholarly publishing, not least because of the constant growth in the number of papers™ — 
or at least they cannot do so within the confines of traditional journal publishing. 


For librarians — who have until now borne the brunt of the serials crisis, and who joined the OA 
movement in the expectation that it would solve their budgetary problems — inevitable 
disappointment lies ahead,® as it does for all those OA advocates who assumed that OA would be 
cheaper. In fact as indicated, some observers believe that OA will turn out to be a more expensive 
method of publishing in the long run. 


This suggests that for research institutions OA simply changes the question from: “How can we 
afford to pay the constantly rising subscription costs that publishers are demanding”, to “How can 
we afford the constantly rising APC prices?” No longer is it a question of, “How can we ensure our 
scientists have access to all the research being produced by other institutions?”, but “How can we 
ensure our scientists have sufficient funds to enable them to publish their research?” 


In short, OA appears destined simply to replace the serials crisis with what one might term an 
author crisis. 


For some authors this could have very serious implications. Those, for instance, based in the 
developing world could find themselves even further disenfranchised from the international 
research community. This point was made in 2008 by Raghavendra Gadagkar of the Centre for 
Ecological Sciences at the Indian Institute of Science in Bangalore. In a letter to Nature Gadagkar 
argued that author-side fees publishing, “does more harm than good in the developing world”. 








He explained: “Page charges may be waived” for authors who cannot afford to pay, but a model 
that depends on payment by authors can afford only a few such waivers. And why should anyone 
want to survive on charity? The argument that it is the granting agency and not the author that 
pays does not wash either. If anything, the playing field for grants is even more uneven.” 


°° As Nature’s Declan Butler put it, “To the average viewer, it must have been perplexing stuff.” 

* As Springer CEO Derk Haank said to me recently, “[T]he reality is that our journals are growing in volume 
by 6% to 7% per year. We have been doing all that is possible over the last couple of year, and will continue to 
do so to ensure that our price increases are lower than the volume increases. But not increasing our prices is not 
an option in the long term.” 

® On the other hand, if the financial responsibility moves from the library to another cost centre within the 
institution some librarians might rejoice that it is no longer their responsibility to pay for scholarly 
communication. 

° Keith Jeffery, Director IT and International Strategy for the Science and Technology Facilities Council at the 
Rutherford Appleton Laboratory (RAL), for instance, believes that OA will prove more expensive than the 
subscription model — certainly for research-intensive institutions. Calculations done at RAL a few years ago, 
he says, demonstrated as much. 

°’ Most OA publishers promise to waive APC costs for researchers in the developing world. However, these 
waivers have been somewhat controversial, and can clearly be withdrawn, or reduced whenever a publisher 
chooses. 
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Gadagkar concluded, “Page charges make extra difficulties for authors, while the old problems 
associated with peer review persist. They could be disastrous for the underdeveloped world, 
encouraging people to remain as consumers (readers), rather than to become producers (authors) 
of knowledge.” 


As promised by PLoS in 2004, we are surely on the threshold of a new era in scholarly 
communication. But how many researchers will be able to afford the entrance fee? 


Author-side fees OA was intended to usher in a revolutionary new way of disseminating research — 
one, as PLoS put it that would ensure “everyone who has access to a computer and an Internet 
connection will be a keystroke away from our living treasury of scientific and medical knowledge.” 


With that revolution now in full flow, however, it turns out that — improved access aside — OA is 
failing to resolve a number of grave problems confronting scholarly communication; rather it 
appears to be amplifying them. While PLoS can rightly boast that it is significantly increasing the 
number of OA papers available, critics argue that it is driving down the quality of those papers, 
and enabling publishers to embed their traditionally high profit levels into the new publishing 
environment — to the point where the research community will sooner or later no longer be able to 
communicate its research effectively. 


As the responsibility for paying the costs shifts from readers to authors, and prices continue to rise, 
some authors will find they simply cannot afford to publish their work.” If that were to happen, 
then the victory that OA supports are claiming for PLoS will prove to have been a pyrrhic victory. 


The death of importance 


So what does the story of PLoS tell us about the likely future of scholarly publishing? The first clue 
lies in the fact that the PLoS ONE model is beginning to be cloned by subscription publishers, not 
just BMJ but others as well. Last December, for instance, Sage announced the launch of a new 
multidisciplinary journal for humanities scholars. Like PLoS ONE Sage Open will be very broad, 
covering “the full spectrum of the social and behavioural sciences and the humanities”. 


And like PLoS ONE, Sage Open reviewers will be instructed to ask a simpler question, and to reach 
an easier decision, when evaluating papers. As the web site puts it, “SAGE Open evaluates the 
scientific and research methods of each article for validity and accepts articles solely on the basis 
of the research. This approach allows readers greater access and gives them the power to 
determine the significance of each article through SAGE Open's interactive comments feature and 
article-level usage metrics.””” 


Similar services have also recently been announced by the American Institute of Physics (A/P_ 
Advances)’' and even Nature (Scientific Reports). 


Nature’s press release announcing Scientific Reports said that the new journal, “will not set a 
threshold of perceived importance for the papers that it publishes; rather, Scientific Reports will 
publish all papers that are judged to be technically valid and original.” 


Like PLoS ONE and like Sage Open, Scientific Reports will also offer article-level metrics, and will 
encourage post-publication commentary. As the release put it, “To enable the community to 





% As we have argued, quality is being driven down not just by PLoS ONE, but by the growing number of 
copycats keen to exploit the business opportunity that the PLoS ONE model provides. 

® While the APC costs of PLoS ONE are lower than some, the sheer number of papers it is publishing, and the 
apparently limited amount of work put into the reviewing process, would certainly suggest it is over-priced, as 
argued earlier. And as we are about to see, it has also sparked a gold rush that is pushing prices up further. 

7 PLoS also pioneered the use of article-level metrics in 2009. 

™ The APC for AIP Advances is $1,350, and the journal’s pre-publication peer review process “focuses on 
whether the manuscript is technically correct and original. Concepts of ‘timeliness’, ‘significance’, or 
‘importance’ are evaluated by the community post publication through the implementation of web 2.0 
commenting and ranking tools.” 
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evaluate the importance of papers post-peer review, the Scientific Reports website will include 
most-downloaded, most-emailed, and most-blogged lists. All research papers will benefit from 
rapid peer review and publication, and will be deposited in PubMed Central.” ”” 


And like PLoS ONE, the APC for both AIP Advances and Scientific Reports will initially be set at 
$1,350. We should however note that the fee for Scientific Reports will increase to $1,700 in 2012 
— demonstrating once again the inflationary forces that drive the scholarly communication market. 





OA advocates were quick to point out that Scientific Reports represents a breath-taking U-turn on 
the part of Nature. Where a few years ago it was deriding bulk-publishing and denigrating cash 
cows it is now embracing them — fearful, presumably, of missing the OA gold rush. Critics dubbed 
the new service “Nature ONE”. 


But who can blame Nature? If Hannay is right to argue that by subsidising its flagship journals PLoS 
has blocked other publishers from developing top-tier OA journals it is only rational for Nature to 
move into the lower-tier publishing market, particularly as that sector appears to offer greater 
profitability.” 


Of course, one could argue with Hannay’s analysis, but it does explain Nature’s decision, and it 
surely increases the likelihood that the quality of published research will continue to fall in coming 
years, with OA being seen as a rapid — but low-quality — publishing option for researchers. 
Moreover as we indicated, by offering cascading peer review services subscription publishers can 
hope to ameliorate the threat posed by OA publishers. 


Our conclusion must therefore be that, however laudable PLoS’ aims (and the aims of the OA 
movement at large), and however desirable it is to make research freely available, PLoS failed to 
appreciate that the greatest problem confronting scholarly communication today is not access but 
affordability. 


After all, if there were no affordability problem there would be no access problem; and as we have 
discovered, OA does not guarantee affordability.” Indeed, as indicated, some fear that it will 
exacerbate the affordability problem, and reduce many scientists to passive consumers of research 
information, rather than active producers. 


In reflecting on the launch of Scientific Reports, Cameron Neylon recently predicted “the future of 
most scholarly publishing will be in publication venues that place no value on a subjective 


x,” 


assessment of ‘importance’. 


Again, whether this would be good or bad depends on one’s point of view. If it meant adopting a 
more realist approach to peer review (the Emperor has no Clothes) some will welcome it. 


If, on the other hand, bulk OA publishing with light review becomes the norm, but publishers 
overcharge for the minimal service they provide, it would be bad news for the research community. 
It would also continue to perpetuate the myth that peer review offers more than it does in reality. 
And it would create a very unfair system, not just for those in the developing world. 





” Note that Scientific Reports talks in terms of blog commentary rather than the provision of comment features 
on its own site. 

7 As an anonymous consultant told Nature in 2008: “BioMed Central knows well that much of the journal 
middle order is more profitable than the great brands because of the lower editorial costs and the cheaper 
marketing costs for bundles of journals. I suspect that PLoS ONE is a result of learning the same lesson.” 
“Tam aware that the discussion in this document is not particularly relevant to self-archiving, or Green OA. 
But here we are discussing Gold “author-side fees” OA, not Green OA — which assumes that the access 
problem can be resolved without changing the current system, or even necessarily migrating from subscription 
to OA journals. As such, it could be argued that Green OA doesn’t address the more serious problems 
confronting scholarly publishing today. On the other hand, one could make a case that it would allow the 
research community to transition to an OA environment in a way that resolved both the access and the 
affordability problems — in so far as it might force publishers to downsize: see 


http://poynder. blogspot.com/2009/03/open-access-who-would-you-back.html). 
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As one disgruntled grad student commented recently: “My problem with Scientific Reports is that 
authors have to pay such an outrageous fee ($1,350 per article) to have their peer reviewed article 
published. This is when | realised that it doesn't matter whether or not | would be ashamed to 
publish in something like Scientific Reports because as a grad student | couldn't friggin’ afford it! 
Apparently, this fee ultimately goes to help fund the publishers’ alpha journals. So people who do 
omega work have to pay exorbitant fees to have their omega work published in an omega journal, 
to help fund the publication of alpha work by alpha researchers. So the alpha keep getting alpha- 
ier while the omega keep getting omega-ier....” 





If this is a true picture of the future, it is no surprise that its critics have come to describe PLoS as 
a “squandered opportunity”. An opportunity, as Anderson put it on The Scholarly Kitchen last year, 
“sacrificed at the altar of open access, of a radicalism blunted into tradition, of audacity 
channelled down the path of least resistance”. 


Anderson added, “It seemed possible that PLoS might be the group to reimagine scholarly 
communication — from peer-review to publication practices to form and function. Advocates 
claimed that their aspirations extended beyond merely creating an alternative economic model for 
publishing ... [but then] ... it went old school, publishing a good traditional journal initially and 
then worrying about traditional publisher concerns like marketing, impact factor, author relations, 
and, of course, the bottom line.” 


In short, PLoS is now charged with having joined the system it was intended to overthrow, and 
becoming the dragon it was meant to slay.” 


On the other hand, if it — along with its clones — was indeed to trigger systemic collapse the 
revolution it predicted might yet be realised, with a more rational system rising from the ashes of 
the old. 


That radical change is coming now seems certain. The quality crisis has reached the point where 
scientists are concluding that, rather than being the handmaiden of science, peer review is its 
scourge. 


Last October, for instance, neurologist Martin Raff said to The Scientist, “Publishing has become a 
nightmare. Particularly for young scientists, who can spend a year or two doing the experiments 
that referees demand. | think this is a terrible thing, because referees’ experiments in my view can 
be dangerous. One, because you’ve put the student or postdoc into a position where they know 
that, if they get the results the referee wants, their paper will get into the journal and their career 
is on its way. If they don’t, they won't get it into a high impact journal and they’re in real trouble. 
That is just putting enormous pressure for selecting your data and cutting corners. So it’s very 
dangerous.” 


Meanwhile, the affordability problem is reaching its own tipping point. Due to the global financial 
crisis it is probably only a matter of time before the research community is forced to say to 
publishers, “Enough is enough; we can no longer pay what you are demanding’. 


As equity analyst Claudio Aspesi pointed out to me last June, publishers have “built a business 
model predicated on annual revenue increases supported by the steady launch of new titles to 
justify the increases. It worked well (at least for them) as long as the librarians were able to find 
the money (regardless of their growing agitation over the increasing spending). It cannot go on if 
the libraries’ budgets continue to be under pressure.””° 


In short, both the quality and the affordability problems are coming to a head. 





® Indeed both PLoS Biology and PLoS Medicine offer a print subscription costing between $365 and $415 ($45 
for a single issue). 

7° Aspesi talks in terms of subscriptions but, as we’ve seen, whether the costs are paid by means of a 
subscription or an APC currently appears to make no difference in terms of affordability. 
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Publish then filter 


So what is the answer? That is really a topic for another day. We should note, however, that Raff’s 
solution sounds very like PLoS ONE. “In the best of all possible worlds,” he told The Scientist, “what 
you would like is for all papers to be published, just making sure that the conclusions are 
reasonable given the observations. And then retrospectively assess the impact and importance of 
the paper.” 


Assuming Raff envisages a system still based on pre-publication review, would not his solution take 
us down the same road as that travelled by PLoS? 


Given PLoS ONE’s inability to attract meaningful post-publication commentary, and its failure to 
resolve the affordability problem, Raff’s proposal does not seem radical enough. As we saw, in 
donning the jewels and ermine of a publisher PLoS appears to have been side-tracked and absorbed 
by the system it was meant to challenge — a victim of some kind of “regulatory capture” perhaps. 





Former BMJ editor Richard Smith, however, may have the answer. Smith’s starting point is the 
escalating quality problem, not affordability. But his proposal could solve both problems in one. 


Smith argues that peer review is now so broken we need to completely rethink the scholarly 
journal. On the BMJ blog in October, for instance, he argued, “The job of journals should not be to 
publish original research but rather to assess critically the research that is published in full on 
databases. Find the comparatively few studies that matter, test them to destruction, and place 
them in their full context.” 


Elsewhere Smith has put it this way: “For journal peer review the alternative is to publish 
everything and then let the world decide what is important. This is possible because of the 
internet, and Charles Leadbeater has illustrated how we have moved from a world of ‘filter then 
publish’ to one of ‘publish then filter’ and a world of ‘I think’ to one of ‘We think’.” 





Smith added, “The problem with filtering before publishing, peer review, is that it is an 
ineffective, slow, expensive, biased, inefficient, anti-innovatory, and easily abused lottery: the 
important is just as likely to be filtered out as the unimportant. The sooner we can let the ‘real’ 
peer review of post-publication peer review get to work the better.” 


In other words, Smith believes that we should abandon pre-prepublication review all together and 
“Instead, let people publish and let the world decide. This is what happens anyway in that what is 
published is digested with some of it absorbed into ‘what we know’ and much of it never being cited 
and simply disappearing.” 


How would such a system work? 


One could envisage a model, for instance, in which researchers were mandated to post all their 
papers (as preprints) in their institutional repository’” — or in a central repository like arXiv or 
PubMed Central. In most cases, this would be the end of the publication process, as only those 
papers deemed to be sufficiently ground-breaking, important, or otherwise worthy of greater 
attention, would be picked out and published in so-called overlay journals.”® These papers would in 
most cases not be published until their significance had been established (a reversal of the PLoS 
ONE model in fact). 


This would also address the problem of researcher reluctance to engage in post-publication review, 
as it would be the task of the editors of overlay journals to identify interesting papers — aided no 
doubt by the constant chatter on blogs, and on social networking sites, where the cutting edge 
analysis and reviewing now seems to take place. 





” As proposed by advocates of Green OA in fact. 
78 Te. journals that harvest papers from multiple online sources in order to create a virtual publication. 
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The editors of overlay journals would then have the papers they had selected reviewed — possibly 
in ways not very dissimilar to how more picky journals do it today, or more likely using a model 
like that pioneered by Faculty of 1000 (but on a pre-publication as opposed to a post-publication 
basis)’. 


Papers not selected for review (the vast majority) would continue to reside in the repositories they 
had been deposited in (living what one might call a “dark archive”) — still discoverable via search 
engines, but not peer reviewed. 


Smith’s proposal would by its very nature resolve the access problem. And since most papers would 
never be formally peer reviewed publishing costs would plummet. 


As it happens, this is exactly the kind of model envisaged by the founders of LiquidPublications, an 
EU-funded project looking at ways of creating new kinds of overlay journals. As University of Trento 
Professor Fabio Casati explained when the project was launched: “Suppose | want to create a 
journal on, say, interesting findings in peer review, that | want to use to collaborate with my 
research group and my peers. | will go fish for interesting papers that are on the Web. People don’t 
submit to my journal, they just post to their webpage or to an archive. | find the paper and include 
it in my journal. Everybody can do this.” 








Peer review purists will of course throw up their arms in horror. But as we’ve seen peer review is 
deeply flawed and getting worse. Casati puts it this way: “We’ve studied this and found that peer 
review doesn’t work, in the sense that there seems to be very little correlation between the 
judgement of peer reviewers and the fate of a paper after publication. Many papers get very high 
marks from their peer reviewers but have little effect on the field. And on the other hand, many 
papers get average ratings but have a big impact.” 


We should note that Casati’s model is not so different from Varmus’s 1999 E-Biomed proposal, 
although perhaps more radical. It is also more radical than arXiv, which still assumes that most 
papers posted to it will also be published in a traditional journal (and so go through traditional peer 
review).°° Nevertheless, Casati’s proposal is inherent in both. One is, therefore, inclined to suggest 
that PLoS might have been better to stick to its knitting and not become a publisher. 


The most important thing 


The fundamental issue that PLoS appears to have overlooked is that all publishers — not excluding 
PLoS, and not excluding learned societies — are so focused on maximising their income that they 
have become blind to the true nature of the crisis confronting scholarly communication.*" 


While publishers are constantly seeking out new ways of making money from the research 
community, the research community is having to accept that unless the costs of scholarly 
communication begin to fall rapidly something is going to break. And it is hard to see how this 
conflict of interest can be resolved within the confines of a journal-based system utilising 
traditional pre-publication review. 


” As Wikipedia explains, “Faculty of 1000 is a website for researchers and clinicians that provides ratings of 
and commentary on scientific research papers. The service acts as a filter, identifying and evaluating the most 
significant articles from biomedical research publications. A peer-nominated 'Faculty' of scientists and clinicians 
rate the articles they read and explain their importance.” F1000 is itself not without its critics. As Bruce Alberts, 
editor in chief of Science and a professor of biochemistry at the University of California at San Francisco, told 
The Chronicle of Higher Education in January, F1000 is prone to dominance by particular individuals within 
subfields, limiting its usefulness as an arbiter of quality. 

*° But not all: In 2002, for instance, the Russian mathematician Grigori Perelman bypassed the peer-review 
system and posted a landmark paper to the online repository, arXiv. Perelman later won the Fields Medal for his 
contribution to the Poincare conjecture, one of mathematics’ oldest and most puzzling problems. 

*' Last year PLoS launched a new service called PLoS Currents. This currently has no APC, but PLoS reserve 
the right to charge one at a later date. It is not clear what the long-term plan is for PLoS Currents, but currently 
it operates pre-publication peer review, and has already attracted criticism and controversy. We should note that 
PLoS responded to this criticism in a conservative rather than radical way. 
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It may be, however, that PLoS editors at least are beginning to reach the same conclusion as Smith. 
In January, for instance, Neylon told Nature that so far as he was concerned it made much more 
sense to “publish everything and filter after the fact”. Neylon also suggested that the logical place 
for post-publication review to occur is on social networking platforms like Twitter, rather than 
journal web sites.® 


Academic editor-in-chief of PLoS Biology Jonathan Eisen appears to be of the same mind, asking 
Nature rhetorically, “Who in their right mind is going to log on to the PLoS ONE site solely to 
comment on a paper? | guarantee that there are more comments on Twitter about a PLoS paper." 


Intriguingly, we should note, Richard Smith is on the PLoS Board of Directors. ® 


Whether we are witnessing the death throes of the traditional journal form is not yet clear, but 
traditional peer review is surely in the process of being strangled in its sleep. 


Let’s finish by going back to the beginning: In declining to answer my questions Binfield appeared 
to suggest that to make a connection between the controversy surrounding the wind setdown paper 
and the larger PLoS story would be inappropriate. | hope | have demonstrated why | disagree with 
this; and | hope | have also explained why | believe that, instead of being a solution to a problem 
(the dire state of scholarly communication), PLoS is in danger of becoming part of that problem. 


For that reason | believe it is both logical and reasonable to draw a straight line from E-Biomed, the 
founding of PLoS, the PLoS ONE business model, the way in which PLoS ONE organises its peer 
review, and the controversy surrounding the wind setdown paper. 


But whatever the reason for Binfield’s reticence, it is surely regrettable that an organisation that 
boasts of its commitment to openness and transparency, and whose business exists courtesy of the 
taxpayer, should decline to answer a list of questions emailed to it by a journalist? 


Should we conclude from this that PLoS has lost its way? If it has, it is probably a consequence of 
the publisher becoming too narrowly focused on “OA at all costs”. Indeed, | am told that that is the 
mantra that PLoS employees repeat to one another when talking at the bar. 


And as we have suggested, the price of “OA at all costs” is likely to be the continuance of an 
unsustainable affordability burden on the research community, and an apparent decline in the 
quality of refereed research. 


Once again we should stress that the story of PLoS is not of one of greedy opportunism or predatory 
intent, but one of unintended consequences. We should likewise stress that Open Access is not only 
desirable but, in today’s scientific environment, essential. PLoS ONE-style OA, however, may not be 
the solution. 


For that reason, it might be timely for PLoS to re-examine its mission, and take a broader approach 
to the crisis it was founded to resolve. 


Time, perhaps, for PLoS founders and managers to gather in the San Francisco café where Varmus 
and Brown met in 1998, and re-appraise their strategy. What better location than a former 
Tassajara Bakery: The Tassajara Zen Mountain Center was founded by the Soto Zen roshi®Shunryu 








*° The key issue, of course, is whether researchers would still be expected to pay $1,350 or more in order to 
have every paper they produce “published” on the Web. 

*° We should also note that Harold Varmus is no longer listed as a member of the PLoS Board of Directors. His 
former role as chairman was taken on by Gary Ward at the end of last year. However, the press release 
apparently failed to point out that Ward was replacing Varmus. Moreover, Varmus is still listed as chairman on 


the about page. 
84 A Roshi is a Zen Master. 
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Suzuki® — who is much-cited for his saying: “The most important thing is to find out what is the 
most important thing”. 


Sitting in San Francisco sipping their caffé latte, PLoS founders and their employees could again 
absorb some Zen wisdom while mulling over three questions: First, “What, in the context of the 
current crisis in scholarly communication, is the most important thing?” Second, are they — 
possibly without realising it — now operating an “OA-at-all-costs strategy?” Third, “If so, do they not 
feel that ‘OA at all costs’ might be a mistaken approach for an organisation wanting to ‘catalyse a 
revolution in scientific publishing”? 


And they should make sure that Richard Smith is present. 


© 2011 Richard Poynder 
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RESPONSE FROM PLoS: 


We were disappointed to see Richard Poynder’s current article about PLoS, but grateful that Richard 
sent us a draft, so that we could prepare a brief response. 


Richard is absolutely correct about one thing — we at PLoS are really committed to open access, and 
we are doing our absolute best to inspire a broader transformation in scientific communication. We 
make no apology for that. We expect to be watched and scrutinized and indeed have been the subject 
of some criticism over the years, but not at the length or with the amount of negativity that we see in 
Richard’s essay. 


Much of the article focuses on PLoS ONE and Richard uses some selected examples (about 5 of 
them) from the more than 17,000 peer-reviewed articles that we’ve published in PLoS ONE to draw 
much broader conclusions about the quality of its content. Although we would be the first to agree 
that PLoS ONE isn’t perfect, neither is any journal, as Richard points out — although not until around 
30 pages into the article. But, just to quote one statistic, is it not more striking that of the 4400 articles 
published in PLoS ONE in 2009 around 55% of them have been cited 3 or more times (Scopus data)? 
The evidence points to the fact that PLoS ONE is attracting a vast amount of high-quality content. 


PLoS ONE is attempting to challenge the conventional model of a journal. The peer review criteria 
for PLoS ONE are focused on rigour, ethical conduct and proper reporting. Reviewers and editors are 
not asked to judge work on the basis of its potential impact. Our argument is that judgments about 
impact and relevance can be left (and might be best left) until after publication, and this argument is 
clearly resonating with the tens of thousands of researchers who work with and support PLoS ONE as 
authors, reviewers or editors. It’s now also resonating with many other non-profit and for-profit 
publishers who are exploring the same model. 








*° Amongst other things, Suzuki popularised Zen Buddhism in the United States. 
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We do not argue that the PLoS ONE approach is the only way to publish research, and indeed we 
view PLoS ONE as just one aspect of a much more fundamental transformation of scholarly 
communication. Another aspect of that transformation is in the assessment and organization of 
research findings, which is currently done using conventional journals. That’s why we have launched 
article-level metrics and PLoS Hubs as new and alternative approaches to post-publication evaluation. 
There will be much more to come from PLoS and many other innovators. 








At several points, Richard’s article uses quotes from staff, press releases and so on that are now 
several years old and misses the point that much has changed even in the short few years since PLoS 
ONE launched. We are learning all the time from PLoS ONE. His frequent quotes from PLoS staff 
also show that we’ve answered many of his questions (including some less than friendly ones) over 
the years. Nevertheless, he places great emphasis on the fact that we declined to answer a set of more 
than 20 detailed and complex questions about general aspects of PLoS ONE, as a follow up to a series 
of exchanges about the peer review process on a particular PLoS ONE article about which there was 
some disagreement. Indeed we posted a comment to try and clarify the issues in light of Richard’s 
questions, and comments from researchers. We were surprised by the number and wide-ranging 
nature of Richard’s subsequent questions about PLoS ONE, and chose not to answer them because we 
felt that the issues surrounding the PLoS ONE article were closed. If Richard had signaled his 
intention to write a lengthy article about the history and status of PLoS at the outset of the exchange, 
our response might have been rather different. 








But the more significant point is that PLoS ONE has evolved since its launch. We did originally 
place a lot of emphasis on ‘commenting’ and ‘rating’ as tools for post-publication assessment, but we 
rapidly realized that much commentary and other activity happens elsewhere. If we could capture this 
activity and add it to the PLoS articles (in all our journals), that could be a powerful approach to post- 
publication assessment, and could also be used to filter and organize content. Thus, the article-level 
metrics project was born. The PLoS ONE editorial and publishing processes are also under constant 
review and revision as the journal’s size and complexity has grown, and we post some updated 
general information about these processes on the PLoS ONE web site. 





Another theme in Richard’s article is whether PLoS ONE represents value for money. PLoS Journals 
are an ecosystem and they all contribute to the whole, both financially and to PLoS’ reputation and 
brand. Looked at in isolation, PLoS ONE, as well as the Community Journals (PLoS Genetics, PLoS 
Computational Biology, PLoS Pathogens and PLoS Neglected Tropical Diseases), all make a positive 
financial contribution to PLoS. They help to support PLoS Biology and PLoS Medicine, as well as 
the development of the journal websites and other important initiatives such as article-level metrics, 
PLoS Hubs, PLoS Currents and our work on advocacy. From the PLoS authors’ perspective, 
wherever they publish in PLoS their work will reach anyone with an interest in it, and their work will 
be stamped with a brand that is associated with social change, innovation and quality. There’s much 
more to value than the direct costs of publishing a single article. 


And a final point on value. Publishing in the conventional system is estimated to cost the academy 
around $4500 per article. What PLoS (and for that matter BioMed Central, Hindawi, Co-Action, 
Copernicus and other successful open-access publishers) is showing is that high-quality publishing 
can be supported by publication fees that are substantially less than the costs of the conventional 
system. 





There is a revolution in the making, and despite the wealth of support that we are seeing, it won’t be 
comfortable for everyone. We need constructive criticism, but also some optimism and creativity to 
make it work. There are grounds for hope that we’ ve moved beyond the antagonism that has 
characterized many discussions around open access. There really is a lot to celebrate. 
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